CasperJS:Submit login-form with evaluate() method

Sometimes the in previous post described method of logging in to a site with casperjs  fails. That could be caused by some restrictions set server-side. Then it might be worth to try the evaluate-call in the login-script:

Example:


var casper = require('casper').create(
{
      logLevel: "info",
      verbose: "true",
      onPageInitialized: function() {
                                                 console.log('Page has been loaded successfully');
                                                 }
});

casper.userAgent('Mozilla/5.0 (compatible; Windows NT 5.0) ' );

casper.viewport = {width: 1366, height: 768};

url = 'http://www.page/login.com';

casper.start(url);

/* Next step is to fill in the credentials and submit the form
Collect id-attributes for username and password elements from source code of login-page.
*/   

casper.then(function() {
console.log('will now log in..');

this.evaluate(function() {

                                   document.getElementById("id-username").value = "username";
                                   document.getElementById("id-password-here").value = "password"
                                   document.getElementById("id-SignIn-button").click();
                                   });

/* If document.getElementById("id-signin-button").click() fails
 try document.querySelector("selector-ofSignIn-button").click();
*/

});

// wait 1 second, then take screenshot
casper.wait(1000, function() {
console.log('now taking screenshot');
casper.capture('photo1.png');
});

casper.run();

Examples of submitting login-forms using CasperJS

One of the standard use-cases of CasperJS is automation of a user logging in to a webpage.

The login-system requires CasperJS to handle:

- Filling in the username and password field
- Submitting the login-form
- Storing cookies received by the server - This is carried out automatically by CasperJS
- Resending cookies after every new HTTP-request( automatically carried out by CasperJS)

Example 1: Using this.sendKeys:

var casper = require('casper').create({

    pageSettings: {

        loadImages: false,//The script is faster when this field is set to false

        loadPlugins: false,

        userAgent: 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36'

    }

});

  var x = require('casper').selectXPath;

casper.start(url-loginpage, function() {

this.sendKeys("#name_username_form_element", "username");

this.sendKeys("#name_password_form_element", "password");

});

// proceed to click submit using XPATH-expression

casper.thenClick(x(' INSERT XPATH EXPRESSION HERE'));

casper.wait(2000,function() {

casper.capture('screenpicture.png');

}),   // wait some time for page to load and take screenshot

casper.run();

XPATH-expression is easily found inspecting the login form with tools such as Chrome-developer

tool or the Firebug extension in Mozilla Firefox.

Note that sometimes it may work to skip the XPATH expression and instead use:

 

casper.then(function() {

this.clickLabel('TITLE-Login_Button');

});

 

Example 2 : Using casper.fill

 

The syntax is then :

 

 

 

casper.fill('selector', {
    'nameOfFormElement' : 'yourInput'
}, submit);

 

 

An example form with selector form#loginform and elements username and password is then filled out as follows:

 casper.fill('form#signInForm', {

    'username' : 'mynamehere',
    'password' : 'difficultpasswordhere'
}, true);

 

In the above expression the submit button is set to true, so it will automatically submit the credentials.

Sometimes it may not work due to some serverside setting. Then at first try to set to 'false' above

and use the .click() method afterwards, i.e add:

 

 

casper.then(function(){
        this.click("selector-signin-button");
    });

 

 

 

 

 

Basic browsing in casperjs

Suppose we want to browse a site and retrieve the page title using casperjs.

First step is to require and create a new casper instance:

var casper = require('casper').create(
{   verbose: true,
    logLevel: 'debug'                              //logLevel parameter can also be 'error', 'info','warning'
    pageSettings: {
                           userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/53"

}

} );


The create() method will then create and return an instance of the casper class.
 Above further optional settings to the create method has been added for debugging, as well
 as a specified useragent.

Second step is to start casper and browse the target site. This is done using the
created casper instance and with it calling the start method:

casper.start(url, function() {// add code here as needed});

The "function()" above denotes a function that may be carried out once the page has been loaded

So let's assume we want to visit google.com:

casper.start(http://google.com);

To browse the google site and print its title:


casper.start(http://www.google.com, function() {
this.echo(this.getTitle(), 'INFO');
});


Messages can be printed in the following range of styles:
'INFO', 'ERROR', 'WARNING', 'COMMENT'

At the end call the run() method with the created casper object:

casper.run();


The resulting code will then be:


 var casper = require('casper').create(
{   verbose: true,
    logLevel: 'debug',
    pageSettings: {
                           userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/53"

}

                                      
} );


casper.start(http://www.google.com, function() {
this.echo(this.getTitle, 'INFO');
});

casper.run();



The above code is saved in a .js file and can be executed by running
casperjs filename.js

Installing SlimerJS-lightweight edition

SlimerJS is similar to PhantomJs, a scriptable browser, except that it runs on top of Gecko, the browser engine of Mozilla Firefox (specifically, version 31), instead of Webkit, and is not yet truly headless.

Since SlimerJS  uses the version of firefox passed in the SLIMERJSLAUNCHER environment variable, edge builds of Firefox can be used. That can be useful for testing and experimenting with modern web-functionality, which is not yet present in PhantomJS.

Since I already have Firefox installed, i will install SlimerJS Lightweight edition from
https://slimerjs.org/download.html:

First download the lightweigt edition 0.9. 6  .zip file to the Downloads directory, then:

cd Downloads
mkdir slimedir
unzip slimerjs-0.9.6.zip -d slimedir


Then open the .bashrc file in a texteditor and at the end insert:

export PATH=$PATH:/Downloads/slimedir/slimerjs-0.9.6

export PATH=$PATH:/Downloads/slimedir/slimerjs-0.9.6/chrome/icons/default

Then reboot.

Basic examples
Get options:


slimerjs --help


A casperjs scriptfile ,here denoted file.js, can be run in context of slimerjs via:

casperjs file.js --engine=slimerjs

From slimerjs.org, an example.js file is:

var webpage = require('webpage').create();
webpage
  .open('http://somewhere') // loads a page
  .then(function(){ // executed after loading
    // store a screenshot of the page
    webpage.viewportSize =
        { width:650, height:320 };
    webpage.render('page.png',
                   {onlyViewport:true});
    // then open a second page
    return webpage.open('http://somewhere2');
  })
  .then(function(){
    // click somewhere on the second page
    webpage.sendEvent("click", 5, 5,
                        'left', 0);
    slimer.exit()
  });

 

The above is run via:

slimerjs example.js

Installing PhantomJS and CasperJS

PhantomJS is a scripted headless browser, based on the WebKit engine. It is used for automating web-page interactions and provides a JavaScript API which enables automation, taking screenshots, simulating user behavior such as submitting forms, clicking links, etc.

Although PhantomJS contains the above utilities on its own, it is easier to use together with
CasperJS  which provides a more intuitive way to script the browser-webpage workflow.

Installing PhantomJS 

cd /usr/local/share
wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-i686.tar.bz2
tar xjvf phantomjs-2.1.1-linux-i686.tar.bz2


Make a symlink and point it to the package , then make symlink in /usr/local/bin to the binary in
/usr/local/share/phantomjs :


ln -s /usr/local/share/phantomjs-2.1.1 /usr/local/share/phantomjs

ln -s /usr/local/share/phantomjs/bin/phantomjs /usr/local/bin/phantomjs



The version of PhantomJS on your system can now be obtained via:


phantomjs --version


Installing CasperJS 

 cd  /usr/local/share

git clone git://github.com/n1k0/casperjs.git
cd casperjs
ln -s /usr/local/share/bin/casperjs /usr/local/bin/casperjs

The installed casperjs version can now be shown via:

casperjs --version

For more details see:
http://phantomjs.org/
http://casperjs.org/

Using the nslookup tool to examine DNS and Mailservers

NSLOOKUP is a tool that is available both on windows and in most Linux distributions. It can be used in settings such as quering DNS-servers for records such as A records (ip-address(es) of server(s) that hosts the domain), MX records(mailexchangeservers), do reverse lookups etc.

Observe that MX records is relevant when someone wants to send a mail to a an emailaddress associated with a domain. The sender's email-client will have to resolve the IP address of the domains mailserver, which is in the MX record of its DNS-server.

NSLOOKUP could thus be useful for troubleshooting situations where a domain has problems receiving emails, i.e to confirm that the domain actually has an MX record and that the MX record is pointed to the correct IP address.

Note that when running:

 nslookup domainname


NSLOOKUP will assume domainname is on the local network and thus will try to resolve the domainname using the internal DNS-server. This will fail but nslookup will proceed to query an external
nameserver that will present the non-authoritative answer obtained.

Example:


C:\Users\jo>nslookup www.google.com
Server:  Myisp.lan
Address:  192.168.2.1:53

Non-authoritative answer:
Name:    www.google.com
Addresses:  82.147.54.27, 82.147.54.21, 82.147.54.16, 82.147.54.28
          82.147.54.15, 82.147.54.23, 82.147.54.25, 82.147.54.26, 82.147.54.24
          82.147.54.19, 82.147.54.18, 82.147.54.17, 82.147.54.29, 82.147.54.22
          82.147.54.22


To use nslookup interactively and set other DNS-server than your own, enter nslookup shell ,
specify the relevant DNS-server-ip(Google DNS), specify type(MX-record) and target(google.com):

nslookup                          
server 8.8.8.8                      
set type=mx                         
google.com
Server:  google-public-dns-a.google.com
Address:  8.8.8.8

Non-authoritative answer:
google.com      MX preference = 50, mail exchanger = alt4.aspmx.l.google.com
google.com      MX preference = 20, mail exchanger = alt1.aspmx.l.google.com
google.com      MX preference = 40, mail exchanger = alt3.aspmx.l.google.com
google.com      MX preference = 10, mail exchanger = aspmx.l.google.com
google.com      MX preference = 30, mail exchanger = alt2.aspmx.l.google.com



There is also the possibility to make queries directly without first entering nslookup shell::

C:\Users\jo>nslookup -query=mx google.com
Server:  Myisp.lan
Address:  192.168.2.1:53
Non authoritative answer::
google.com      MX preference = 30, mail exchanger = alt2.aspmx.l.google.com
google.com      MX preference = 20, mail exchanger = alt1.aspmx.l.google.com
google.com      MX preference = 50, mail exchanger = alt4.aspmx.l.google.com
google.com      MX preference = 40, mail exchanger = alt3.aspmx.l.google.com
google.com      MX preference = 10, mail exchanger = aspmx.l.google.com

google.com      nameserver = ns1.google.com
google.com      nameserver = ns3.google.com
google.com      nameserver = ns4.google.com
google.com      nameserver = ns2.google.com
aspmx.l.google.com      internet address = 74.125.136.26
alt1.aspmx.l.google.com internet address = 74.125.200.26
alt2.aspmx.l.google.com internet address = 74.125.23.26
alt3.aspmx.l.google.com internet address = 173.194.72.26
alt4.aspmx.l.google.com internet address = 74.125.25.26
ns1.google.com  internet address = 216.239.32.10
ns2.google.com  internet address = 216.239.34.10
ns3.google.com  internet address = 216.239.36.10
ns4.google.com  internet address = 216.239.38.10




Query for Google nameservers:
nslookup -query=ns google.com
Server:  Myisp.lan
Address:  192.168.2.1:53

Non-authoritative-answer:
google.com      nameserver = ns3.google.com
google.com      nameserver = ns4.google.com
google.com      nameserver = ns2.google.com
google.com      nameserver = ns1.google.com

ns1.google.com  internet address = 216.239.32.10
ns2.google.com  internet address = 216.239.34.10
ns3.google.com  internet address = 216.239.36.10
ns4.google.com  internet address = 216.239.38.10



Same result is also obtained via
nslookup -type=ns google.com

Retrieving lost .jpg-files

Scalpel is a file carving software that in most linux distributions can be installed from the repositories via

sudo apt-get install scalpel

According to the standard definition of .jpg format these files may have two different signatures,
 i.e two different setups of headers:


\xff\xd8\xff\xe0\x00\x10

And:

\xff\xd8\xf1\xe1


For both of the above variants of the header, there is a trailing footer at the end of each file:

\xff\xd9


Note that '\x' before each hexadecimal number is the standard marker for hexadecimal notation.

To be convinced about the above, just open a random .jpeg file in a hexeditor and observe the header and footer.

The above info will then be used to make a custom scalpel configuration file, here called
customscalpel.conf:

jpg y 20000000 \xff\xd8\xff\xe0\x00\x10 \xff\xd9
jpg y 20000000 \xff\xd8\xf1\xe1 \xff\xd9


The configuration file is thus customized to make scalpel look in a specified filesystem for .jpg
 files up to 20Mb. The second row enables scalpel to not discard .jpeg files with the other
allowed signature.

If the target drive that is supposed  to be examined with scalpel is /dev/sda, then scalpel is launched via


sudo scalpel -c customscalpel.conf -o someoutfilename /dev/sda


Here the -o flag  generates an outputfile where the results are collected.

Note that upon installing scalpel a standard configuration file, scalpel.conf, is automatically placed in

/etc/scalpel


To use it, just open the config-file in a text-editor and remove the '#' sign in front of relevant fileformats.

However, many formats are not present in the default installed configuration file.