Automation software

[StepOne]

Hi,

I'm looking for a piece of software to automate manual tasks. It would need to be able to log in to a webpage and download a list of information into Excel. I then want to step through each entry in that list, extracting more detailed information from the website. I would need to be able to direct the software to enter information into a text box on the webpage, click a 'Select' or 'Data' button on the webpage, then copy information on the resulting page to the clipboard, and paste into the excel sheet. I might also have to 'look' for text on a page - e.g. find where it says 'Weather' and then click the button immediately below it. That sort of thing.

I've used things like this in the past for software testing, and I guess that some of these test tools would do the job, but it's really not test tools that we are after.

Robotask comes up with a few of the searched, and it looks like it might do the job, but I wanted to ask on here if anyone has any personal experience or recommendations.

Thanks,
StepOne

[johnhemming]

Selenium might do the job. It is testing software. I looked at it a while ago (over a year) and it seemed quite flexible.

[uspaul666]

Selenium would definately be able to do that and recently started offering a “headless” mode that means the process can run on a server without a display screen. Apache poi would be able to handle the excel spreadsheet. But someone’s going to have to learn java or python which isn’t necessarily trivial.

[UncleEbenezer]

You have two options for that kind of automation. Find a tool that does it, or script it yourself.

I strongly recommend the latter. Once you've scripted one or two such tasks, every future one is essentially 'free'. That is to say, you can do it much quicker and easier than finding another tool for the next task, and then figuring out how to work with it. And no cost - ever. A general-purpose scripting language has easy library functions for everything: in your case, you'd be looking for an http-client (to fetch and interact with a web page), an HTML parser (to extract data), and an excel API, or maybe simple export-as-CSV.

Never heard of "selenium". I'd be suspicious of anything too packaged: it may solve this problem, but I'd expect it to have just as much learning curve as (say) Perl or Python, and leave you needing to go through all the seeking and learning next time you have a different task to automate.

[pochisoldi]

Hi,

I'm looking for a piece of software to automate manual tasks. It would need to be able to log in to a webpage and download a list of information into Excel. I then want to step through each entry in that list, extracting more detailed information from the website. I would need to be able to direct the software to enter information into a text box on the webpage, click a 'Select' or 'Data' button on the webpage, then copy information on the resulting page to the clipboard, and paste into the excel sheet. I might also have to 'look' for text on a page - e.g. find where it says 'Weather' and then click the button immediately below it. That sort of thing.

I've used things like this in the past for software testing, and I guess that some of these test tools would do the job, but it's really not test tools that we are after.

Robotask comes up with a few of the searched, and it looks like it might do the job, but I wanted to ask on here if anyone has any personal experience or recommendations.

Thanks,
StepOne

Cygwin with the appropriate scripting support, and support for cron would be able to do that in the background dumping the output to .csv or text file which could be read or imported into an Excel sheet.
Bad news is that you'd have to create it for yourself using your favorite scripting language (e.g. Perl, python, php etc).

PochiSoldi

[servodude]

I use python
- more precisely splinter (https://pypi.org/project/splinter/) which wraps up selenium in python
- a quick read at the splinter tutorial can give you an idea of what you can do (https://splinter.readthedocs.io/en/latest/tutorial.html)

On windows I installed python via anaconda (https://www.anaconda.com/download/) which also includes the Spyder IDE (which i find bearable)
- then once I have a script I drop it in the windows task scheduler

It's worth a look:
- there's a little bit of learning curve
- but a lot of power and a large community

have fun
- sd

[Wmnr]

I've used Autoit in the past. Runs on Windows and it's free

[StepOne]

Thanks for all the replies.

We are going to go down the 'develop it yourself' route. I have done similar stuff before with Excel VBA in the early days of data scraping on the Motley Fool, so it should be do-able. Will look into the Python/Selenium thing though - looks interesting...

Cheers,
StepOne

[scotia]

I have done similar stuff before with Excel VBA in the early days of data scraping on the Motley Fool,

I was going to suggest VBA. My choice would probably be Access (with VBA) to gather, process and store intermediate data, and to construct the desired Spreadsheet output.

[eepee]

Your first task must be to find out if you can scrape the data!

Copy and transfer out, from the page of interest, manually and see if you end up with the informaion you want.

Some sites - especially the ones that supply an API for extracting information, can make data scraping virtually impossible.

If you want to see what I mean, try it on an event on Betfair.

Regards,
ep