Yahoo price scrape has failed

[johnstevens77]

I cannot update the prices on HYPTUSS, I just have the above message and to try again but to no avail.

john

[mike]

Same for me using the Simple Yahoo Price Scrape in LibreOffice. I'm on v1.3 - I downloaded the newer v1.5 and received the same response.

[mc2fool]

Not sure if HYPTUSS uses the same API mechanism as I do, but digging into a little more detail I'm getting a return status of 401 Unauthorized and a JSON return of: {"finance":{"result":null,"error":{"code":"Unauthorized","description":"Invalid Crumb"}}}

Checking logs (don't ask!) it seems to have been happening sporadically since Saturday and with a vengeance in the past couple of hours.

P.S. I use, e.g., https://query1.finance.yahoo.com/v7/finance/quote?symbols=BP.L,SHEL.L&fields=regularMarketPrice,regularMarketOpen,regularMarketDayHigh,regularMarketDayLow,regularMarketTime,gmtOffSetMilliseconds,longName,shortName,marketState,symbol

[mc2fool]

Single ticker historical query still seems to work.

https://query1.finance.yahoo.com/v7/finance/download/SHEL.L?period1=1650396785&period2=1681932785&interval=1d&events=history&includeAdjustedClose=true

[mc2fool]

Ok, well on a look through live HTTP headers and a very cursory test, it seems the thing to do is to first call:

https://query2.finance.yahoo.com/v1/test/getcrumb

which returns something like: 6esNRzDT6Ag, and then to bung &crumb=6esNRzDT6Ag on the end of the query.

Guess I now know what I'm doing tomorrow, sigh...

Edit: Ugh...there's cookie(s) involved too.

[mike]

A few years back, this would happen semi-regulary, but I have not seen it for some time. It would generally self-correct.

I've just tried again using the Simple Yahoo Price Scrape in LibreOffice, and received this error message in place of the simple failed message previously

com.sun.star.uno.RuntimeException: Error during invoking function UpdateYahooPricePy in module vnd.sun.star.tdoc:/1/Scripts/python/SimpleYahooPriceScrape.py (<class 'OverflowError'>: timeout doesn't fit into C timeval
File "C:\Program Files\LibreOffice\program\pythonscript.py", line 915, in invoke
ret = self.func( *args )
File "vnd.sun.star.tdoc:/1/Scripts/python/SimpleYahooPriceScrape.py", line 226, in UpdateYahooPricePy
File "vnd.sun.star.tdoc:/1/Scripts/python/SimpleYahooPriceScrape.py", line 117, in getHtml
)

I've just copied and pasted - the hanging ")" is as displayed in the error message box

[Arborbridge]

I cannot update the prices on HYPTUSS, I just have the above message and to try again but to no avail.

john

I ran the prices late afternoon today, and it worked fine.

How frustrating! Hopefully it will sort itself out tomorrow.

[Sankarani]

Thanks for posting. Same problem here. At least I now know it's not my computer. Fingers crossed for tomorrow.

[eventide]

Yahoo have seemingly built cookie based auth into anything calling the v7 api. It’s possible to work around. A right pia in excel/vba but should be less hassle if backend is Python requests (just auth a session with friendly headers to get the cookie in place, then use session.get(…) instead of requests.get(…)

[kiloran]

I'll look at this when I get a chance, but it won't be soon. Wife in hospital for the past 7 weeks and now mum in hospital 300 miles away.

--kiloran

[torata]

I'll look at this when I get a chance, but it won't be soon. Wife in hospital for the past 7 weeks and now mum in hospital 300 miles away.

--kiloran

Hope things are not too hectic for you.
Speaking for everyone - "Yes, look after your priorities. We're happy to wait."

torata

[spiderbill]

I'll look at this when I get a chance, but it won't be soon. Wife in hospital for the past 7 weeks and now mum in hospital 300 miles away.--kiloran

Very sorry to hear that. I do hope that both recover well and as quickly as possible.

Do not worry about anything else. Family comes first.
We'll manage, there are workarounds. (I've just set up a Google sheet that I can copy prices from into both HYPTUS in LibreOffice and my other Excel spreadsheet that uses eventide's method.)

best wishes
Spiderbill

[Itsallaguess]

I cannot update the prices on HYPTUSS, I just have the above message and to try again but to no avail.

It looks to be working fine again now, so hopefully just a temporary Yahoo blip similar to the type of short-term issues we've seen in the past.

Cheers,

Itsallaguess

[mc2fool]

Having spent a fair amount of time on this today, I can only say, bar stewards!

A google reveals that this "crumb and cookie" validation has appeared before, although never in exactly this form. For the record, just in case, what I found worked was adding &crumb=<crumb> to the URL and setting the "A1" cookie. FWIW, values I found working today where:

crumb = "IRwR2mo35hf";
cookies = "A1=d=AQABBH8kQWQCEEw-kMPgQf2APjVJgDo-3aAFEgABCAFwQmRqZPbPb2UB9qMAAAcIfSRBZN_UZR8&S=AQAAAksvGy-50TZ3oM-KlxD3GxE";

The trouble was finding them. The crumb was easy enough (having visited yahoo first to get the cookies in place), but the A1 cookie doesn't come from simply fetching finance.yahoo.com, and indeed I looked through the return headers of the huge plethora of GETs that page issues and I couldn't find a Set-Cookie for A1 (although may be a case of hiding in plain sight amongst the myriad of them), and I suspect it's set dynamically by JavaScript.

(I found it was the vital one by elimination; simply deleting each finance.yahoo.com cookie until it no longer worked...)

[Itsallaguess]

Having spent a fair amount of time on this today, I can only say, bar stewards!

As an aside, and just in case this does ever become a longer-term issue, I've been using the v10 Yahoo API modules for quite a while now, for scraping other share-related attributes via JSON, and it's probably worth noting here that whilst the v7 API price-scrape issue raised on this thread was failing over the past couple of days, the v10 API modules continued to work fine, so it's probably worth me noting a single-share 'v10 API price-module' URL here, for Vodafone as an example, just to give a 'Plan B' option for future reference if it's ever needed -

https://query2.finance.yahoo.com/v10/finance/quoteSummary/vod.l?modules=price




Source - https://query2.finance.yahoo.com/v10/finance/quoteSummary/vod.l?modules=price

Whilst it's not quite as swift as the comma-separated shares list option that's available via the current v7 API process, it's still a rather rapid scrape when throwing single-share EPIC's at it via Excel VBA, and on a 30-odd share portfolio the above v10 JSON price-scrape will grab all prices in just over 2 seconds, so maybe a worthwhile option if it's ever required...

Cheers,

Itsallaguess

[mc2fool]

As an aside, and just in case this does ever become a longer-term issue, I've been using the v10 Yahoo API modules for quite a while now...

Oooh, do you have a pointer to the API docs? And do I understand correctly that you can't fetch data for multiple stocks in one GET?

[Itsallaguess]

As an aside, and just in case this does ever become a longer-term issue, I've been using the v10 Yahoo API modules for quite a while now...

Oooh, do you have a pointer to the API docs?

And do I understand correctly that you can't fetch data for multiple stocks in one GET?

No pointer to the docs, sorry, but I've got a list of other v10 modules that might be useful or interesting for you, other than the price one captured in the earlier v10 API URL -

assetProfile
summaryProfile
summaryDetail
esgScores
price
incomeStatementHistory
incomeStatementHistoryQuarterly
balanceSheetHistory
balanceSheetHistoryQuarterly
cashflowStatementHistory
cashflowStatementHistoryQuarterly
defaultKeyStatistics
financialData
calendarEvents
secFilings
recommendationTrend
upgradeDowngradeHistory
institutionOwnership
fundOwnership
majorDirectHolders
majorHoldersBreakdown
insiderTransactions
insiderHolders
netSharePurchaseActivity
earnings
earningsHistory
earningsTrend
industryTrend
indexTrend
sectorTrend

I've not had success with all of them, but just listed here if you want to have a play with some.

Obviously I can't vouch for the returned data, so please do carry out enough sanity-checking on any areas of interest to make sure it's safe to use for anything...

Regarding your second point around multiple tickers, I've not had any success in getting anything other than data for a single EPIC returned with the v10 API URL shown earlier, and copied below using the VOD price module for completeness, but to be honest, for what I want to do it's fast enough for me on a single-share basis, even when looking at fairly long lists of EPICs...

https://query2.finance.yahoo.com/v10/finance/quoteSummary/vod.l?modules=price

I think you can stack modules, if that helps, so here's the VOD price and financial data module-data stacked in a single URL -

https://query2.finance.yahoo.com/v10/finance/quoteSummary/vod.l?modules=price%2CfinancialData

Cheers,

Itsallaguess

[mc2fool]

Hmmm ... is there a V8 and/or V9 API?

[mc2fool]

This appears to be a comprehensive list, albeit a mix of V1, V6/7, V8, V10 and V11 APIs!

https://cryptocointracker.com/yahoo-finance/yahoo-finance-api

[johnstevens77]

It is working now.

john