Basic browsing in casperjs

Suppose we want to browse a site and retrieve the page title using casperjs.

First step is to require and create a new casper instance:

var casper = require('casper').create(
{   verbose: true,
    logLevel: 'debug'                              //logLevel parameter can also be 'error', 'info','warning'
    pageSettings: {
                           userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/53"

}

} );


The create() method will then create and return an instance of the casper class.
 Above further optional settings to the create method has been added for debugging, as well
 as a specified useragent.

Second step is to start casper and browse the target site. This is done using the
created casper instance and with it calling the start method:

casper.start(url, function() {// add code here as needed});

The "function()" above denotes a function that may be carried out once the page has been loaded

So let's assume we want to visit google.com:

casper.start(http://google.com);

To browse the google site and print its title:


casper.start(http://www.google.com, function() {
this.echo(this.getTitle(), 'INFO');
});


Messages can be printed in the following range of styles:
'INFO', 'ERROR', 'WARNING', 'COMMENT'

At the end call the run() method with the created casper object:

casper.run();


The resulting code will then be:


 var casper = require('casper').create(
{   verbose: true,
    logLevel: 'debug',
    pageSettings: {
                           userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/53"

}

                                      
} );


casper.start(http://www.google.com, function() {
this.echo(this.getTitle, 'INFO');
});

casper.run();



The above code is saved in a .js file and can be executed by running
casperjs filename.js

Installing SlimerJS-lightweight edition

SlimerJS is similar to PhantomJs, a scriptable browser, except that it runs on top of Gecko, the browser engine of Mozilla Firefox (specifically, version 31), instead of Webkit, and is not yet truly headless.

Since SlimerJS  uses the version of firefox passed in the SLIMERJSLAUNCHER environment variable, edge builds of Firefox can be used. That can be useful for testing and experimenting with modern web-functionality, which is not yet present in PhantomJS.

Since I already have Firefox installed, i will install SlimerJS Lightweight edition from
https://slimerjs.org/download.html:

First download the lightweigt edition 0.9. 6  .zip file to the Downloads directory, then:

cd Downloads
mkdir slimedir
unzip slimerjs-0.9.6.zip -d slimedir


Then open the .bashrc file in a texteditor and at the end insert:

export PATH=$PATH:/Downloads/slimedir/slimerjs-0.9.6

export PATH=$PATH:/Downloads/slimedir/slimerjs-0.9.6/chrome/icons/default

Then reboot.

Basic examples
Get options:


slimerjs --help


A casperjs scriptfile ,here denoted file.js, can be run in context of slimerjs via:

casperjs file.js --engine=slimerjs

From slimerjs.org, an example.js file is:

var webpage = require('webpage').create();
webpage
  .open('http://somewhere') // loads a page
  .then(function(){ // executed after loading
    // store a screenshot of the page
    webpage.viewportSize =
        { width:650, height:320 };
    webpage.render('page.png',
                   {onlyViewport:true});
    // then open a second page
    return webpage.open('http://somewhere2');
  })
  .then(function(){
    // click somewhere on the second page
    webpage.sendEvent("click", 5, 5,
                        'left', 0);
    slimer.exit()
  });

 

The above is run via:

slimerjs example.js

Installing PhantomJS and CasperJS

PhantomJS is a scripted headless browser, based on the WebKit engine. It is used for automating web-page interactions and provides a JavaScript API which enables automation, taking screenshots, simulating user behavior such as submitting forms, clicking links, etc.

Although PhantomJS contains the above utilities on its own, it is easier to use together with
CasperJS  which provides a more intuitive way to script the browser-webpage workflow.

Installing PhantomJS 

cd /usr/local/share
wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-i686.tar.bz2
tar xjvf phantomjs-2.1.1-linux-i686.tar.bz2


Make a symlink and point it to the package , then make symlink in /usr/local/bin to the binary in
/usr/local/share/phantomjs :


ln -s /usr/local/share/phantomjs-2.1.1 /usr/local/share/phantomjs

ln -s /usr/local/share/phantomjs/bin/phantomjs /usr/local/bin/phantomjs



The version of PhantomJS on your system can now be obtained via:


phantomjs --version


Installing CasperJS 

 cd  /usr/local/share

git clone git://github.com/n1k0/casperjs.git
cd casperjs
ln -s /usr/local/share/bin/casperjs /usr/local/bin/casperjs

The installed casperjs version can now be shown via:

casperjs --version

For more details see:
http://phantomjs.org/
http://casperjs.org/

Using the nslookup tool to examine DNS and Mailservers

NSLOOKUP is a tool that is available both on windows and in most Linux distributions. It can be used in settings such as quering DNS-servers for records such as A records (ip-address(es) of server(s) that hosts the domain), MX records(mailexchangeservers), do reverse lookups etc.

Observe that MX records is relevant when someone wants to send a mail to a an emailaddress associated with a domain. The sender's email-client will have to resolve the IP address of the domains mailserver, which is in the MX record of its DNS-server.

NSLOOKUP could thus be useful for troubleshooting situations where a domain has problems receiving emails, i.e to confirm that the domain actually has an MX record and that the MX record is pointed to the correct IP address.

Note that when running:

 nslookup domainname


NSLOOKUP will assume domainname is on the local network and thus will try to resolve the domainname using the internal DNS-server. This will fail but nslookup will proceed to query an external
nameserver that will present the non-authoritative answer obtained.

Example:


C:\Users\jo>nslookup www.google.com
Server:  Myisp.lan
Address:  192.168.2.1:53

Non-authoritative answer:
Name:    www.google.com
Addresses:  82.147.54.27, 82.147.54.21, 82.147.54.16, 82.147.54.28
          82.147.54.15, 82.147.54.23, 82.147.54.25, 82.147.54.26, 82.147.54.24
          82.147.54.19, 82.147.54.18, 82.147.54.17, 82.147.54.29, 82.147.54.22
          82.147.54.22


To use nslookup interactively and set other DNS-server than your own, enter nslookup shell ,
specify the relevant DNS-server-ip(Google DNS), specify type(MX-record) and target(google.com):

nslookup                          
server 8.8.8.8                      
set type=mx                         
google.com
Server:  google-public-dns-a.google.com
Address:  8.8.8.8

Non-authoritative answer:
google.com      MX preference = 50, mail exchanger = alt4.aspmx.l.google.com
google.com      MX preference = 20, mail exchanger = alt1.aspmx.l.google.com
google.com      MX preference = 40, mail exchanger = alt3.aspmx.l.google.com
google.com      MX preference = 10, mail exchanger = aspmx.l.google.com
google.com      MX preference = 30, mail exchanger = alt2.aspmx.l.google.com



There is also the possibility to make queries directly without first entering nslookup shell::

C:\Users\jo>nslookup -query=mx google.com
Server:  Myisp.lan
Address:  192.168.2.1:53
Non authoritative answer::
google.com      MX preference = 30, mail exchanger = alt2.aspmx.l.google.com
google.com      MX preference = 20, mail exchanger = alt1.aspmx.l.google.com
google.com      MX preference = 50, mail exchanger = alt4.aspmx.l.google.com
google.com      MX preference = 40, mail exchanger = alt3.aspmx.l.google.com
google.com      MX preference = 10, mail exchanger = aspmx.l.google.com

google.com      nameserver = ns1.google.com
google.com      nameserver = ns3.google.com
google.com      nameserver = ns4.google.com
google.com      nameserver = ns2.google.com
aspmx.l.google.com      internet address = 74.125.136.26
alt1.aspmx.l.google.com internet address = 74.125.200.26
alt2.aspmx.l.google.com internet address = 74.125.23.26
alt3.aspmx.l.google.com internet address = 173.194.72.26
alt4.aspmx.l.google.com internet address = 74.125.25.26
ns1.google.com  internet address = 216.239.32.10
ns2.google.com  internet address = 216.239.34.10
ns3.google.com  internet address = 216.239.36.10
ns4.google.com  internet address = 216.239.38.10




Query for Google nameservers:
nslookup -query=ns google.com
Server:  Myisp.lan
Address:  192.168.2.1:53

Non-authoritative-answer:
google.com      nameserver = ns3.google.com
google.com      nameserver = ns4.google.com
google.com      nameserver = ns2.google.com
google.com      nameserver = ns1.google.com

ns1.google.com  internet address = 216.239.32.10
ns2.google.com  internet address = 216.239.34.10
ns3.google.com  internet address = 216.239.36.10
ns4.google.com  internet address = 216.239.38.10



Same result is also obtained via
nslookup -type=ns google.com

Retrieving lost .jpg-files

Scalpel is a file carving software that in most linux distributions can be installed from the repositories via

sudo apt-get install scalpel

According to the standard definition of .jpg format these files may have two different signatures,
 i.e two different setups of headers:


\xff\xd8\xff\xe0\x00\x10

And:

\xff\xd8\xf1\xe1


For both of the above variants of the header, there is a trailing footer at the end of each file:

\xff\xd9


Note that '\x' before each hexadecimal number is the standard marker for hexadecimal notation.

To be convinced about the above, just open a random .jpeg file in a hexeditor and observe the header and footer.

The above info will then be used to make a custom scalpel configuration file, here called
customscalpel.conf:

jpg y 20000000 \xff\xd8\xff\xe0\x00\x10 \xff\xd9
jpg y 20000000 \xff\xd8\xf1\xe1 \xff\xd9


The configuration file is thus customized to make scalpel look in a specified filesystem for .jpg
 files up to 20Mb. The second row enables scalpel to not discard .jpeg files with the other
allowed signature.

If the target drive that is supposed  to be examined with scalpel is /dev/sda, then scalpel is launched via


sudo scalpel -c customscalpel.conf -o someoutfilename /dev/sda


Here the -o flag  generates an outputfile where the results are collected.

Note that upon installing scalpel a standard configuration file, scalpel.conf, is automatically placed in

/etc/scalpel


To use it, just open the config-file in a text-editor and remove the '#' sign in front of relevant fileformats.

However, many formats are not present in the default installed configuration file.

Simple FM-receiver with RTLSDR-dongle and gnuradio-companion

GNU Radio is a free software development toolkit which makes the task of signal processing the input from sources such as the cheap 10$-rtlsdr dongles easier.

Bundled together with  GNU Radio is gnuradio-companion which is a python  code generator tool.
Gnuradio-companion is used to build 'flowgraphs'. When the flowgraph is constructed and 'compiled', python code is generated that creates and connects the blocks in the flowgraph. These blocks then step by step 'clean' and demodulate the incoming signal .

Below is the flowgraph I built using gnuradio-companion. A RTLSDR dongle is used as signal source to receive on a frequency containing FM-radio:

Note that the rtlsdr-dongle is just receiving the signal and not demodulating it. The signal is fed into gnu-radio companion for processing and demodulation.

Info about the flowgraph

RTLSDR source has samplerate set to 2.0e6Mhz/s

Ch0 frequency in the source-block is associated with the wxgui-slider variable where max is 110Mhz,min 90 Mhz,default 97.796Mhz.
Outgoing signals are then fed to a wxgui FFT-sink, a waterfall sink and a rational resampler.The FFT(Fast Fourier Transform)-sink and waterfall sink are mainly there to get a quick visual view of any signals coming in from the signal source.

In the rational resampler the decimation rate is set to 4 to reduce the signal by 1/4. This is done to prevent the CPU from overloading. To further clean up the signal it is fed into a lowpass-filter,where the cutoff frequency is set to 100000Hz and the transition width is set to 1000000.

Then the signal is sent to a wideband-FM-demodulator block,WBFM-receive, where the quadrature
rate is set to 500000 and decimation to 1.

Afterwards another resampler-block with interpolation 48 and decimation 500 prepare the signal for next block which is the audio sink. Finally the signal is sent to the computer-sound card which in gnuradio-companion is represented by an audio-sink.Most computer soundcards operate with samplerate 48Khz so it is essential that this is set in the audio sink.

Note: Sometimes the property device-name in the audio-sink must be defined.
Usually it can be left empty but if no sound is produced  try to set device-name as 'pulse' incase your OS uses pulseaudio to handle sound.

If alsa is used on your OS, device-name parameter of audio sink could be set as 'hw:0,0' or 'plughw:0,0'

When the flowgraph is generated(i.e compiled) and executed FM-radio should be heard from
sound card accompanied by an output similar to:

Building multimon-ng from source and capture FMSFSK-signals

I noticed that the kali linux 2.0 repository version of multimon-ng does not include the FMSFSK demodulation option.

FMSFSK demodulation in multimon-ng is used to demodulate German Funkmeldesystem-signals.

To get the complete multimon-ng version, first install its dependencies:


apt-get install qt4-qmake  libpulse-dev  cmake  libx11-dev  pulseaudio


Then clone and build multimon-ng from source:

git clone https://github.com/EliasOenal/multimonNG.git

cd multimonNG

mkdir build && cd build

qmake ../multimon-ng.pro

make && make install

ldconfig



Then connect the rtlsdr-dongle to the computer and hook it up to a decent antenna.

The rtlsdr-dongle's ppm-value should also be identified via tools such as kalibrate for more reliable results.

When an appropriate frequency is found, pipe the sound from rtl_fm to multimon-ng for demodulation:


rtl_fm -f 431.865e6 -p -2 -l 100 | multimon-ng -a FMSFSK -a AFSK1200 -t raw /dev/stdin