Javascript has become more and more popular over the last few years, especially since the introduction of the runtime known as NodeJS.
This step-by-step guide will explain everything involved in scraping currency data with NodeJS.
Prerequisites
This guide is aimed at software developers with a certain level of Javascript experience. However, we will do our very best to describe every step in a detailed way, so that beginners can also follow.
To get the project started, we will run the following code:
npm init
npm i axios
npm i xml-js
npm init will create a new npm project. Before running this command, move to a directory of your choice.
npm i axios adds axios to our project. Axios is a promise-based HTTP client. It is simple to use but comes with a
ton of flexibility at the same time.
npm i xml-js adds xml-js to our project. We will use this package to convert scraped XML to JSON. Some people choose to work with XML directly, but JSON has gained popularity over the past few years.
Choosing a Data Source
The ECB provides an XML-feed covering 31 currencies. The data is updated daily, which should be perfectly fine for a hobby project of our kind.
Let us open the feed with our browser. What we will see is the latest currency data for the 31 most important currencies in XML format.
Scraping ECB Currency Data
Now that we have our data source let us start scraping the data.
First, create a new file called scraper.js inside our project folder.
Using the promise syntax of Axios, our code will look similar to this:
const axios = require('axios')
axios
.get('https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml')
.then((response) => {
console.log(response)
})
.catch((error) => {
console.log(error)
})
Let’s run the code from our CLI by running:
node scraper.js
Assuming the request was successful, the response object contains a key called data.
This includes the data of our interest. You will notice the variable type of data is still a string.
Hence, we must first parse the data to XML (and then to JSON).
This is the point where the second package xml-js comes into play. It allows us to pass the XML as a string and converts everything into JSON.
Implementing this into our code will change our snippet to the following:
const axios = require('axios')
const convert = require('xml-js')
/**
* @param {string} data raw scraped data
* @returns {JSON}
*/
function convertToJson(data) {
return JSON.parse(convert.xml2json(data, {compact: true, spaces: 2}))
}
axios
.get('https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml')
.then((response) => {
let parsedJSON = convertToJson(response.data)
let currencies = parsedJSON['gesmes:Envelope'].Cube.Cube.Cube
console.log(currencies)
})
.catch((error) => {
console.log(error)
})
As you can see, we have added a function called contertToJson(), which converts the XML string to a JSON string and returns it as a parsed JSON object.
To be honest, the object contains the desired data but comes along in a pretty messy and hard-to-read shape. Therefore, we focus on the inner part of the object by running the selector: parsedJSON['gesmes:Envelope'].Cube.Cube.Cube
Of course, we could now go ahead and take this further by remoting the _attributes key and casting the rate value into a float variable. However, since we want to focus on the underlying concepts of getting the data, we will stick with the object the way it is.
We are left with a JSON object that contains all relevant information:
[
{
"_attributes":{
"currency":"USD",
"rate":"1.0570"
}
},
{
"_attributes":{
"currency":"JPY",
"rate":"137.90"
}
},
...
]
Scraping Exchange Rates from Yahoo Finance
Since the ECB provides the currency data in a very easily accessible XML-Feed, let us look at another potential data source where the data is part of an HTML page and can be accessed with a javascript client request.
The first part, scraping the data, stays precisely the same as for our previous example - we simply adjust the target URL to https://finance.yahoo.com/currencies:
axios
.get('https://finance.yahoo.com/currencies')
.then((response) => {
console.log(response.data);
})
This will show us the HTML body of the page containing the currency exchange rate data. Compared to our previous example, it is not enough to simply convert the retrieved string into JSON.
This time, we actually have to parse the data we are looking for. For this, we are going to use a library called jsdom. To install it, we are going to run yarn add jsdom.
Once the install has finished, we add it to the import section of our code snippet:
const axios = require('axios')
const jsdom = require("jsdom") // new
const { JSDOM } = jsdom // new
Now, let’s take a look at our final scraper/parser. We will be walking through the code step-by-step afterward:
const axios = require('axios')
const jsdom = require("jsdom")
const { JSDOM } = jsdom
axios
.get('https://finance.yahoo.com/currencies')
.then((response) => {
let dom = new JSDOM(response.data)
let pairRows = dom.window.document.querySelectorAll('#yfin-list table tbody tr')
let results = []
pairRows.forEach(pairRow => {
rate = {
'name': pairRow.querySelector('td[aria-label="Name"]').textContent,
'price': parseFloat(pairRow.querySelector('td[aria-label="Last Price"]').textContent.replaceAll(',', '')),
'change': pairRow.querySelector('td[aria-label="Change"]').textContent,
'percentChange': pairRow.querySelector('td[aria-label="% Change"]').textContent,
}
results.push(rate);
})
console.log(results)
})
.catch((error) => {
console.log(error)
})
Parsing the Data with JSDOM
The first part should look familiar; hence we will not talk about what is happening.
let dom = new JSDOM(response.data) is the part where it becomes interesting. This line brings the HTML string into a DOM that allows us to run selector queries.
The page contains a lot of code, and we want to focus on the table that contains the exchange rate information.
Therefore, the following selector will collect every row (every currency pair) inside the table:
let pairRows = dom.window.document.querySelectorAll('#yfin-list table tbody tr')
Since we want to parse each currency pair explicitly, we loop over the array of results, parse each column of our interest and push it into a results array that we are going to log in our console:
pairRows.forEach(pairRow => {
rate = {
'name': pairRow.querySelector('td[aria-label="Name"]').textContent,
'price': parseFloat(pairRow.querySelector('td[aria-label="Last Price"]').textContent.replaceAll(',', '')),
'change': pairRow.querySelector('td[aria-label="Change"]').textContent,
'percentChange': pairRow.querySelector('td[aria-label="% Change"]').textContent,
}
results.push(rate);
})
At first sight, parsing the specific columns looks a little tricky, as they do not have specific or reliably looking ids or class names. However, each column has an attribute called “aria-label” that describes the cell’s content in a very descriptive way.
The 'price' column is worth mentioning at this point, as we want to convert the parsed string into a float. We do this by wrapping the selector with the function parseFloat.
However, before doing that, we remove all commas from the string, as this would distort the results (e.g. 14,630.0000 would turn into 14 and not 14630).
Results & Summary
When running the script from our terminal, we end up with a nicely formatted array full of scraped currency data:
I hope you liked the tutorial! If you need reliable currency data for over 170 currencies for a fair price, click here.