Quick Tip: HTTP GET with the ESP8266 Thing

The SparkFun ESP8266 “Thing” is one of the cheapest Internet of Things (IoT) platforms available.

SparkFun ESP8266 Thing
Photo courtesy of sparkfun.com

There are some great examples on how to post data to data.sparkfun.com, but we need to modify that code in order to pull data from a website. That is accomplished with the humble HTTP GET request. I put together a quick example that pulls www.example.com and prints it to the serial console. Note that you will need perform some slight modifications to the board (or use Realterm) to see the serial data.

[Updated 11/26/15] Added #include <WiFiClient.h>. Thanks, Naguissa!

 

51 Responses

  1. Hello,

    Thanks for the post. I’m doing a home alarm system and using a GET request to my server to raise alerts. This example is perfect as base to the program.

    One detail: There’re two libraries missing needed to run the sketch:

    #include
    #include

    Cheers.

  2. My webpage is returning JSON which gets printed to the serial console by this section:

    // If there are incoming bytes, print them
    if ( client.available() ) {
    char c = client.read();
    Serial.print(c);
    }

    How can I store this into a variable so that I can use it with a JSON parsing library? I’ve been struggling with this for days…

    1. You will need to create a buffer (probably a byte or char array) and store everything you get from client.read(). You will also need to look for an “End of File” or “End of Stream” character (or set of characters) to let the loop know when you’re done. I’m not sure what this would be from your webpage, so you’ll have to look at the output to determine that. A good example of how this works is the readPage() function in this example: http://bildr.org/2011/06/arduino-ethernet-client/.

  3. my module(olimex esp8266 dev board) LED still blinking not connected to WiFi after upload the sketch several times.

  4. What if I want to set a cookie before accessing a webpage.. That’s because the webpage redirects to some other page without that cookie..

  5. How can I read out the source code of a website? or webdokument
    I have a txt-file on my server with temperature data which I want to display via a esp8266

    1. If you are using a browser on a desktop, you can press ‘F12’ to view the source (HTML). Are you using the ESP8266 as the server, or is the ESP8266 trying to read the text file from another computer?

      1. Yes the ESP8266 is trying to read a text file from a webserver data.MYDOMAIN.com/data.txt

        The text file just look like this:
        0,255,255,0

        1. From a browser on another computer, can you navigate to the site “data.MYDOMAIN.com/data.txt” and see the contents in the browser window? This will let you know if the server is responding to GET requests.

  6. What method would I use to send a variable to a php file using this code?
    I have tried the same way I do with the regular Ethernet.h library but nothing is showing up in my database…
    client.print( “GET /add_data_motion1.php?”);
    client.print(“motion1=”);
    client.print( motion1 );
    client.println( ” HTTP/1.1″);
    client.print( “Host: ” );
    client.println(http_site);
    client.println( “Connection: close” );
    client.println();

    Thanks!

  7. I want to fetch the JSON data from Node.Js Server to ESP8266-01 using http GET method.

    I have used Software serial definition as
    #include
    SoftwareSerial ser(A8,A9); // Connect TX and RX of ESP8266 (A8 for TX A9 for RX) of ATMEGA2560.

    In the code i have written

    ser.print(String(“GET “) + “/details HTTP/1.1/\r\n” +
    “192.168.1.11”+ “\r\n” +
    “Connection: close\r\n\r\n”);

    // Read all the lines of the reply from server and print them to Serial
    Serial.println(“Respond:”);
    while(ser.available()){
    String line = ser.readStringUntil(‘\r’);
    Serial.print(line);
    }

    Serial.println();
    Serial.println(“closing connection”);

    where ser is defined function for ESP8266-01
    192.168.1.11 is Local server IP address from where i have to fetch the JSON data(data and time in my case).

    But i am not able to fetch date and time data.

    1. Are you able to access the JSON data from another client, like your computer?

      Additionally, I’m not super well versed in Arduino’s String object. Try doing a Serial.print with your entire GET request to make sure it looks right. I get the feeling trying to concatenate a String object with strings (arrays of characters) might not work that well.

      1. Yes.I am able to access data from another client.

        Sir ,i will post my query again in more understandable form.

        We have created Node.Js server in our office network for testing purpose.

        Now i am able to send the data to Node.js server that is temperature and humidity values are getting updated to Node.js server.

        Now i am stucked with receiving the data from Node.Js server(date and time from internet ) to ESp8266 and have to dispay on serial monitor of arduino which is not happening .

        Here is the code.

        I have initialized for ESP8266 for ATMEGA 2560 as below shown code and

        used ser.println function,hope using “ser” is correct? 

        #include 

        SoftwareSerial ser(A8,A9);

        Then in the esp8266 function as

        void esp_8266()
        {

        String cmd = “AT+CIPSTART=\”TCP\”,\””; // AT cammand for TCP/IP connection
        cmd += “192.168.1.11”; // node.js Server Ip address
        cmd += “\”,5000″; // HTTP port no.

        ser.println(cmd); // Sending AT cammand
        p=1;
        if(p == 1)
        {
        if(ser.find(“Linked”)) // Serching for responce from ESP8266
        {

        Serial.println(“Connection Made for Receiving data”);
        z=0;
        }

        p=0;
        }

        int deviceid = serialNumber;
        String getStr = “GET /log/345/12/status/”; // /log is the url , 345,12 ,status are parameters which are hardcoded for testing

        getStr += DHT.temperature; //room temperature
        getStr += “/”;
        getStr += DHT.humidity; //room humidity
        getStr += “\r\n\r\n”;

        cmd = “AT+CIPSEND=”; // AT command to send Data at TCP server
        cmd += String(getStr.length()); // send data length
        ser.println(cmd); // Wrighting the AT command to ESP8622

        if(ser.find(“>”)) // Serching for responce to send data from ESP8266
        {

        ser.print(getStr); // Printing the string with sensor data to send on ESP8266 module

        }

        delay (100);

        ser.print(String(“GET “) + “/details HTTP/1.1/\r\n” +
        “192.168.1.11”+ “\r\n” +
        “Connection: close\r\n\r\n”);

        // Read all the lines of the reply from server and print them to Serial
        Serial.println(“Respond:”);
        while(ser.available()){
        String line = ser.readStringUntil(‘\r’);
        Serial.println(line);
        }

        Serial.println();
        Serial.println(“closing connection”);
        }

          1. Since I am not familiar with AT commands on the ESP8266, I recommend looking at another example, like this one. See if that works for you. You’ll have to change the Serial1 to your SoftwareSerial.

      1. hey man, I’m getting this error below. Any idea why?

        In file included from C:\Users\01738\Documents\Arduino\libraries\ESP8266WiFi\src/ESP8266WiFi.h:33:0,

        from C:\Users\01738\Documents\Arduino\ESP8266_ATfromcode2\ESP8266_ATfromcode2.ino:9:

        C:\Users\01738\Documents\Arduino\libraries\ESP8266WiFi\src/ESP8266WiFiType.h:26:19: fatal error: queue.h: No such file or directory

        #include

        ^

        compilation terminated.

        Using library ESP8266WiFi at version 1.0 in folder: C:\Users\01738\Documents\Arduino\libraries\ESP8266WiFi
        exit status 1
        Error compiling for board Arduino/Genuino Mega or Mega 2560.

    1. That seems odd since it’s a fairly basic HTTP page, no SSL or anything. Do you have any more information on the error, such as an HTTP status code? Can you successfully request other non-SSL sites?

      1. Thanks for responding! I’ve tried other sites e.g. http://www.google.com with success as well as http://www.example.com. However this site http://api.usno.navy.mil/easter?year=2016 or this usefull site api.usno.navy.mil/rstt/oneday?date=9/14/2016&coords=47.73N,9.16E&tz=1 respond to an browser request but don’t accept a connection from within the sample program. May be you find the time to test it by your own.Thank you for supporting me.

        here is my minor modifyed code (line 9-12, ine 97ff:
        #include
        #include

        // WiFi information
        const char WIFI_SSID[] = “—“;
        const char WIFI_PSK[] = “—-“;

        // Remote site information
        //const char http_site[] = “www.example.com”;
        const char http_site[] = “http://api.usno.navy.mil/rstt/oneday?date=9/14/2016&coords=47.73N,9.16E&tz=1”; // this one don’t respond
        const char site[] = “http://api.usno.navy.mil/easter?year=2016”; // this one don’t respond -> no server connection however as browser request it works fine!
        //char site[] = “www.google.com”; // this one works fine, “got connection”

        const int http_port = 80;

        const int LED_PIN = 16; //16=red LED 2=blue LED on ESP-12 board

        // Global variables
        WiFiClient client;

        void setup() {

        // Set up serial console to read web page
        Serial.begin(115200);
        Serial.println(); Serial.println();
        Serial.println(“Thing GET Example”);

        // Set up LED for debugging
        pinMode(LED_PIN, OUTPUT);

        // Connect to WiFi
        connectWiFi();

        // Attempt to connect to website
        if ( !getPage() ) {
        Serial.println(“GET request failed”);
        }
        }

        void loop() {

        // If there are incoming bytes, print them
        if ( client.available() ) {
        char c = client.read();
        Serial.print(c);
        }

        // If the server has disconnected, stop the client and WiFi
        if ( !client.connected() ) {
        Serial.println();

        // Close socket and wait for disconnect from WiFi
        client.stop();
        if ( WiFi.status() != WL_DISCONNECTED ) {
        WiFi.disconnect();
        }

        // Turn off LED
        digitalWrite(LED_PIN, LOW);

        // Do nothing
        Serial.println(“Finished Thing GET test”);
        while(true){
        delay(1000);
        }
        }
        }

        // *************** Attempt to connect to WiFi
        void connectWiFi() {

        byte led_status = 0;

        // Set WiFi mode to station (client)
        WiFi.mode(WIFI_STA);

        // Initiate connection with SSID and PSK
        WiFi.begin(WIFI_SSID, WIFI_PSK);

        // Blink LED while we wait for WiFi connection
        while ( WiFi.status() != WL_CONNECTED ) {
        digitalWrite(LED_PIN, led_status);
        led_status ^= 0x01;
        delay(100);
        }

        // Turn LED on when we are connected
        digitalWrite(LED_PIN, HIGH);
        Serial.println(“WiFi Connected”);
        }

        // ************* Perform a HTTP GET request to a remote page
        bool getPage() {

        // Attempt to make a connection to the remote server
        // if ( !client.connect(http_site, http_port) ) {
        delay(500);
        if ( client.connect(site, http_port))
        {Serial.println(“server connected”);
        return true;}
        else
        {
        Serial.println(“no server connection”);
        return false;}

        1. Alright, I got it working. As it turns out, you can’t list the full URL in http_site. It should just be the base site, “api.usno.navy.mil/” (no http://) otherwise the DNS server won’t return an IP address. You then need to add the resource location “/rstt/oneday?date=9/14/2016&coords=47.73N,9.16E&tz=1” right after “GET.” I don’t have a standalone ESP8266, so I couldn’t use your code exactly. Here is what I used on the SparkFun Thing:

          Edit: Code formatting doesn’t seem to work in comments, so I posted it here: https://gist.github.com/ShawnHymel/3b1e2d2649f5dded8f66ed1e9ff8d35c

          1. BINGO!!! Thank you so much Shawn, I got the right answer now. Running on an ESP8266 all you need is to #include and
            #include :-))

            Beeing so happy now I like to send you a littel present. Whats your postal address? Pls supply it to my email account.

            Next step will be decoding/parsing the json file. Any hint how to get started?

  8. Thanks for the example and support! May I ask for help again.
    While setup WiFi connection my scetch – based on your example (thanks again!) – occasionally (app. every third call) end with an Exception (9) …
    I’ve no idea what’s the reason nor how to avoid this :-((
    ——
    char versions_id[] = “ESP__WEB_test_V6”;
    #include
    #include
    #include
    #include

    const char WIFI_SSID[] = “***”; // WiFi information
    const char WIFI_PSK[] = “***”;

    const char http_site[] = “api.usno.navy.mil”; // Remote site information
    const int http_port = 80;

    const int LED_PIN = 16; // global variables & definitions
    WiFiClient client;
    char c; // incomming char from WEB-server
    unsigned long delayMillis; // delay loop time
    unsigned long pause = 20000; // prog paused

    // JSON stuff
    StaticJsonBuffer jsonBuffer; // Memory pool for JSON object tree.
    char json[500]; // JSON object char array, filld with server response
    String head = “”; // HTTP header from server
    int json_i=0; // index for json char array
    int JS_bracket = 0; // counter for JSON { and } brackets

    void setup() {
    Serial.begin(115200); // Set up serial console to read web page
    Serial.println(); Serial.println(versions_id);
    Serial.println(“********************RE-/START******************”);
    pinMode(LED_PIN, OUTPUT); // Set up LED for debugging
    }

    void loop() {
    head = “”; // reset server response string
    JS_bracket = 0; // reset json bracket counter

    connectWiFi(); // Connect to WiFi

    if ( !getPage() ) { // Attempt to connect to website
    Serial.println(“GET request failed”);}
    else {
    Serial.println(“GET request ok”);}

    while (client.connected() ) {
    if ( client.available() ) { // If there are incoming bytes, print them
    c = client.read();
    //Serial.print(c);
    if (JS_bracket == 0) {head=head+c;} // fill in server response
    if (c == ‘{‘) {++JS_bracket;} // ectraxt json object
    if (c == ‘}’) {–JS_bracket; }
    // still to do: skip white space outside “..”
    if (JS_bracket > 0) { //Serial.print(c); // look for matching closing bracket
    json[json_i]=c; json_i++;}; }
    }
    json[json_i]=’}’; // add a closing } bracket
    Serial.println(“stop client”); // If the server has disconnected, stop the client and WiFi
    client.stop(); // Close socket and wait for disconnect from WiFi
    if ( WiFi.status() != WL_DISCONNECTED ) {
    WiFi.disconnect();}
    digitalWrite(LED_PIN, HIGH); // Turn off LED

    // *****process server data here
    Serial.println(“processing server data”);
    // for (int i=0; i<=json_i; i++) {Serial.print(json[i]);} Serial.println(); // control print
    Serial.println("******************pausing****************");
    delayMillis= millis()+pause; // 5 minutes intervall
    do {
    yield();
    } while (millis() < delayMillis);
    Serial.println("***********start after pausing***********");
    // *****end processing server data

    }

    //******************connect WiFi***********************
    void connectWiFi() { // connect WiFi
    byte led_status = 0;
    WiFi.mode(WIFI_STA); // Set WiFi mode to station (client)
    WiFi.begin(WIFI_SSID, WIFI_PSK); // Initiate connection with SSID and PSK
    while ( WiFi.status() != WL_CONNECTED ) { // Blink LED while we wait for WiFi connection
    digitalWrite(LED_PIN, led_status);
    led_status ^= 0x01;
    delay(100);
    }
    digitalWrite(LED_PIN, HIGH); // Turn LED on when we are connected
    }

    //***********************get page***************
    bool getPage() { // Perform an HTTP GET request to a remote page
    if ( !client.connect(http_site, http_port) ) { // Attempt to make a connection to the remote server
    return false;
    }
    client.println("GET /rstt/oneday?date=9/14/2016&coords=47.73N,9.16E&tz=1 HTTP/1.1"); // Make an HTTP GET request
    client.print("Host: ");
    client.println(http_site);
    client.println("Connection: close");
    client.println();
    return true;
    }

  9. some one eats text inside angle brackets

    #include ..ESP8266WiFi.h..
    #include ..WiFiClient.h..
    #include ..ArduinoJson.h..
    #include ..TimeLib.h..

  10. Hi to all

    I´m newbie in arduino and ndemcu, and I have a question.

    I´m testing my nodemcu esp8266 1.0 esp-12E module module with arduino ide and I already made some little projects and all it´s working fine.

    I would like to know if its possible to access a url from nodemcu , i.e.,
    I have a account in voipraider thats allow me to send a sms through this url:

    If I paste this url in any browser, voipraider send a sms to destination_phone.

    My goal is create a sketch in nodemcu to send a sms using this url.

    Its possible ?
    If yes, could you please help me with some code to use in arduino ide (not LUA) ?

    I would be very grateful if you could help me.

    Thanks for your attention

    1. I’m sorry, but I am not familiar with NodeMCU, so I don’t think I’ll be able to help you much. If you are just pasting a URL into a browser, then you are making an HTTP GET request. As long as you can do that with your microcontroller, then you should (in theory) be able to make the SMS request.

  11. Hi to all

    I´m newbie in arduino and ndemcu, and I have a question.

    I´m testing my nodemcu esp8266 1.0 esp-12E module module with arduino ide and I already made some little projects and all it´s working fine.

    I would like to know if its possible to access a url from nodemcu , i.e.,
    I have a account in voipraider thats allow me to send a sms through this url:

    If I paste this url in any browser, voipraider send a sms to destination_phone.

    My goal is create a sketch in nodemcu to send a sms using this url:

    Its possible ?
    If yes, could you please help me with some code to use in arduino ide (not LUA) ?

    I would be very grateful if you could help me.

    Thanks for your attention

Leave a Reply