Danmorgan.net

I needed to iterate through an unknown amount of records, but I need to do it in a non-sequential manner, as to appear that I was not enumerating the data set in its entirety.

So how could someone enumerate every single record, non-sequential, without duplication, indefinably? I modified my Fisher Yates shuffle to do this exact task:

function nonseq($amount,$iter=0,$seed=NULL) {
	if (!$seed) {
		list($usec, $sec) = explode(' ', microtime());
		$seed = (float) $sec + ((float) $usec * 100000);
	}

	srand($seed);

	for($i=0;$i<$amount;$i++) $map[$i] = ($i + 1);

	for($i=(count($map)-1);$i>=0;$i--) {
		$j = @rand(0,$i);
		$tmp = $map[$i];
		$map[$i] = $map[$j];
		$map[$j] = $tmp;
	}
	if ($iter > ($amount)) {
		$base = floor($iter / $amount);
		$niter = $iter - ($base * $amount);
		$prec = nonseq($amount,($niter+1),$seed+1);
		return $prec + ($base * $amount);
	}
	return $map[$iter-1];
}

The code is also located here

This function takes 3 parameters, but two are optional. The first, and required, parameter is increment partition size. The second parameter is the normal sequential pointer. If you do not provide a second parameter then it will default to 0. The third parameter is a random seed.

Logic of the function:
X = Increment Partition Size, Y = Sequential Pointer, Z = Random Seed.
If (X >= Y), results = (FYShuffled number between 0 and X randomed by Z)
If (X < Y ), results = ((FYShuffled number between 0 and X randomed by ( Z+(X/Y) )) + (largest multiple of X that is less than Y))

<?
include("nonseq.php");
echo nonseq(10,50,5);
?>

The above examples first parameter instructs the function to randomize sequential in increments of 10. The second value asks what the 50th sequential iteration would be converted to, and the 5 is a simple random seed. Since the first value is 10, then we can assure that the output will be between 51 and 60, and in my case was : 60. If we run the same function with the sequential iteration set to 51 we get : 54.

So if you were to normally iterate through records as such :

for($i=1;$i<=3000;$i++) {
   $cont = file("http://www.scrape-site.com/sites.php?id={$i}");
   record_info($cont);
}

This example would iterate sequentially ( id=1, id=2, id=3, id=4, id=5...)
However, if you use the nonseq function :

for($i=1;$i<=3000;$i++) {
   $nsid = non_seq(1000,$i,5);
   $cont = file("http://www.scrape-site.com/sites.php?id={$nsid}");
   record_info($cont);
}

This would iterate in a non-sequential manner(id=832, id=272, id=698, id=177...). It would continue and without repeating any numbers indefinably.

So you want to spoof your useragent in your php script that uses CuRL? If your intentions are good then I got the script for you!

Using browser statistics and my weighted random script I made a function that will rerun a user agent based on statistics of the top browsers (IE, FireFox,Chrome,Opera,Safari).

You can download the file here : Random User Agent

Then simply include the file and call your curl_setopt to define your useragent :

include("random-user-agent.php");

//.... some code

curl_setopt($ch, CURLOPT_USERAGENT,random_user_agent());

//.... lots more code

Use with a TOR Proxy to become unstoppable!

Good Luck

TOR, or The Onion Router is a great tool if you wish to do anything anonymously. There are a few pitfalls that we will cover, but first the easy part.

Installing TOR

First download the sourceball at https://www.torproject.org/download/download.html.en#source

extract and before you run configure make sure you have the supporting libraries installed by running :

sudo apt-get install libevent-dev libssl-dev

Then to compile

./configure

make

sudo make install

Simply run tor to start the tor service.

Tor by default runs localhost on port 9050.

You can

man tor

to get more options for the config filee Config file can be built in /usr/local/tor with the following command :

cp /usr/local/etc/tor/torrc.sampl /usr/local/tor/torrc

A good Config settings to have are the following :

NewCircuitPeriod 1
MaxCircuitDirtiness 10
EnforceDistinctSubnets 1

 

This will help renew circuits for each request, so you can jump around IP addresses.

If you are going to use PHP CuRL to connect to this proxie then please remember to set this CuRL Option :

curl_setopt($ch,CURLOPT_PROXYTYPE,CURLPROXY_SOCKS5);
curl_setopt($ch,CURLOPT_PROXY, '127.0.0.1:9050');

 

You now have your own anonymous Socks5 proxy… but there are things to watch out for.

Pitfall

Once you start using Tor you will find this warning in your logs :

Your application (using socks5 to port 80) is giving Tor only an IP address. Applications that do DNS resolves themselves may leak information. Consider using Socks4A (e.g. via privoxy or socat) instead. For more information, please see http://wiki.noreply.org/noreply/TheOnionRouter/TorFAQ#SOCKSAndDNS.

The biggest problem with Tor is the DNS requests. That is, although you can use Tor to anonymize your traffic, you will first send a DNS request untorified in order to get the IP address of the target system. The solution can be to use only IP addresses (unrealistic), or to anonymize your DNS. I will cover this on a later post.