Author Topic: Limit mysql executions/ grabbing information from sites (Read 581 times)

gamerz300 · « **on:** July 17, 2011, 01:28:48 PM »

First:
what i mean is like a db update

i update per second lets say as i know mysql will rape a cpu if i attempt to execute 100 queries large or small at once lol i want them to be executed 1 per x sec kind of thing. im looking at using a loop/if function...

Second:

i want to grab specific information from a site like a name of a object cost etc. and store it into a db (storing is easy enough)
i know of preg_match/all but im wondering if there is another one or no? (i have also seemed to of lost my extended php manual lol) if not one can someone give a example based off of this topic or my profile page lol?

Edit: Answer in 2 hours would be greatly appreciated but not needed

Edit 2: No i wont be using this to steal peoples info i said my profile because i have its code lol

Pinako · « **Reply #1 on:** July 17, 2011, 03:30:22 PM »

1. What I'd do is launch two separate processes or threads, one to pull data from the source and push it into a queue (or deposit it on the filesystem), one to pull data from the queue and push it into the database and sleep occasionally. You don't even need two threads if you don't mind doing the two tasks sequentially or using some additional timing logic, but I'll leave that as an exercise for you.

2. People would tell you that you can't parse HTML using regular expressions, and recommend that you use a real SGML parser (or at least a lenient XML parser). I'd say that if you don't care about robustness, go right ahead with regular expressions.

gamerz300 · « **Reply #2 on:** July 17, 2011, 08:15:52 PM »

cool thanks, even tho on my psp i cant really read it lol

gamerz300 · « **Reply #3 on:** August 08, 2011, 09:31:47 AM »

ok i spent 5 hours yesterday to get it working twice first time i stupidly did not make a copy lol...

getting it to grab and display was easy enough but i seem to kill the script when i add (no matter where) the required db insert... when i get a chance i'll post the code

Code: [Select]

<?php
/* gets the data from a URL */
function get_data($url)
{
  $ch = curl_init();
  $timeout = 5;
  curl_setopt($ch,CURLOPT_URL,$url);
  curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
  curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);
  $data = curl_exec($ch);
  curl_close($ch);
  return $data;
}

$returned_content = get_data('http://URL');
$html = str_replace(array("\r","\n"), "", $returned_content);
$regexp = '#Insert Content Here#';
preg_match($regexp, $html, $matches);
if (count($matches)==0)return false;
$name = $matches[1];
$img = $matches[2];
$alt = $matches[3];
$dis = $matches[4];
$price = $matches[5];
$change1 = $matches[6];
$change2 = $matches[7];
$change3 = $matches[8];
echo "" . $price . "<br>";
echo "" . $change1 . "<br>";
echo "" . $change2 . "<br>";
echo "" . $change3 . "<br>";
echo "" . $name . "<br>";
echo "" . $alt . "<br>";
echo "" . $dis . "<br>";
echo "" . $img . "<br>";

unset($matches);
?>

It works but wont post to any db >.>
i get
"Unknown column 'Name' in 'field list'"
Name is the result of $name

Edit 2:

Also a if with $i++ loop seems to kill it as well... if someone could figure this out for me or give a lending hand on why its not working would be great.

Confuser · « **Reply #4 on:** August 08, 2011, 11:08:58 AM »

Post your insert query and the $i++ loop code

PS: your error means that the column Name does not exist in your table

gamerz300 · « **Reply #5 on:** August 08, 2011, 01:36:58 PM »

i'll post the whole code when i get back on the computer

and i figured what happened with sql i ' ' the colum name lol so thats fixed

i'll either edit this post like above or repost when i get on

Code: [Select]


<?php
include('config.php');
include('opendb.php');

/* gets the data from a URL */
function get_data($url)
{
  $ch = curl_init();
  $timeout = 5;
  $userAgent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)';
  curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
  curl_setopt($ch,CURLOPT_URL,$url);
  curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
  curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);
  $data = curl_exec($ch);
  curl_close($ch);
  return $data;
}
$itemid= 6;
for($i=2; $i<=$itemid; $i++) {
$returned_content = get_data('http://URL?=' . $itemid . '');
$html = str_replace(array("\r","\n"), "", $returned_content);
$regexp = '#Insert Content Here#';
preg_match($regexp, $html, $matches);
if (count($matches)==0){
echo "";
}else{
$name = $matches[1];
$img = $matches[2];
$alt = $matches[3];
$dis = $matches[4];
$price = $matches[5];
$change1 = $matches[6];
$change2 = $matches[7];
$change3 = $matches[8];
echo "" . $price . "<br>";
echo "" . $change1 . "<br>";
echo "" . $change2 . "<br>";
echo "" . $change3 . "<br>";
echo "" . $name . "<br>";
echo "" . $alt . "<br>";
echo "" . $dis . "<br>";
echo "" . $img . "<br>";



unset($matches);
mysql_query("INSERT INTO Item (name, img, alt, dis, price, change1, change2, change3, itemid) VALUES ('$name', '$img', '$alt', '$dis', '$price', '$change1', '$change2', '$change3', '$img');") or die(mysql_error());
	echo " Added! </ br>";
}
}

?>

the $img is the same as id as its a ?=#
Edit sql before or after unset it dont matter... im trying it on xampp before i try on a server.

Confuser · « **Reply #6 on:** August 08, 2011, 03:45:12 PM »

What do you mean "kill" it? As in apache crashes or the browser just loads for like ages and looks like it does nothing?

gamerz300 · « **Reply #7 on:** August 08, 2011, 04:46:34 PM »

kill might not be the proper word...

it comes up blank unless you add something in the blank echo yet 2 should pull a result. (so should 6)

remove the if/else i get
Notice: Undefined offset: 1 in C:\xampp\htdocs\Testing.php on line 30
Keeps doing this to offset: 8 Line 30 = $name = $matches[1];

For a kicker i added _all to preg_match without if/else i get a array with it i still get an array

Confuser · « **Reply #8 on:** August 08, 2011, 05:06:00 PM »

If undefined offset then it means it didnt find a match, double check your regex

gamerz300 · « **Reply #9 on:** August 08, 2011, 05:36:48 PM »

ill check it either later or tomorrow i think its right but i could be wrong lol I was thinking the same but the ?=# for the page's content is all the same but what im grabbing...

gamerz300 · « **Reply #10 on:** August 09, 2011, 11:03:01 AM »

i found something wrong and fixed it but i still get Undefined offset so idk i'' keep looking...

wondering

can i do more regex's? using the same returned_content?

Confuser · « **Reply #11 on:** August 09, 2011, 12:01:58 PM »

print_r($matches)
Without your regex and example content your searching through its difficult to help

gamerz300 · « **Reply #12 on:** August 09, 2011, 05:49:54 PM »

well thanks to print_r($matches) i found out what went wrong idk how but i fixed it (the image with ?= somehow screwed it up...)

~~but now the loop aint working lol the loop is looping 6...~~

lolz at above... stupid me $itemid should not be where it is in the loop it should be $i...

AND.... It works

New:
ummm ok i forget now but how do i add a ' with out botching up the code?

<h2>(.*?)</h2>

its with in the (.*?) that has it...

Confuser · « **Reply #13 on:** August 10, 2011, 04:38:27 AM »

The matched value of (.*?) has a ' in it? Shouldnt make a difference unless ive miss understood your question?

gamerz300 · « **Reply #14 on:** August 10, 2011, 07:10:41 AM »

yea somewhere in there has a '. and apparently it matters >.>

Quote

You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 's blackhole.

however it still grabs all the required info... so idk whats wrong

Oh i used addslashes but it made no difference...

Primefalcon · « **Reply #15 on:** August 10, 2011, 08:44:39 AM »

Anything you want php to treat at face value such as a '

at a backslash before it such as \'

this way php will not try to treat it as a special character

gamerz300 · « **Reply #16 on:** August 10, 2011, 03:15:04 PM »

well the problem is it is with in the (.*?). however i fixed that for the most part(or all i goofed on one line of matches)... i used addslashes to the matches[\#] <<< lolz BB Code...

i used addslashes to the actual regex instead of the actual result from each match (should not make a difference imo but it does)

Edit: with over 3000 needed to be executed is there a way of it real time echoing "added"?

Confuser · « **Reply #17 on:** August 10, 2011, 04:32:08 PM »

ob_start() and ob_flush() should be able to send HTML to the browser whilst processing still. Just make sure gzip is disabled as that is known to cause problems

gamerz300 · « **Reply #18 on:** August 10, 2011, 05:00:13 PM »

cool thanks, however looking at the samples i dont want anymore issues with the script >.< as its running fine and i have it to die at 3000 seconds...

Author Topic: Limit mysql executions/ grabbing information from sites (Read 581 times)

gamerz300

Limit mysql executions/ grabbing information from sites

Pinako

Re: Limit mysql executions/ grabbing information from sites

gamerz300

Re: Limit mysql executions/ grabbing information from sites

gamerz300

Re: Limit mysql executions/ grabbing information from sites

Confuser

Re: Limit mysql executions/ grabbing information from sites

gamerz300

Re: Limit mysql executions/ grabbing information from sites

Confuser

Re: Limit mysql executions/ grabbing information from sites

gamerz300

Re: Limit mysql executions/ grabbing information from sites

Confuser

Re: Limit mysql executions/ grabbing information from sites

gamerz300

Re: Limit mysql executions/ grabbing information from sites

gamerz300

Re: Limit mysql executions/ grabbing information from sites

Confuser

Re: Limit mysql executions/ grabbing information from sites

gamerz300

Re: Limit mysql executions/ grabbing information from sites

Confuser

Re: Limit mysql executions/ grabbing information from sites

gamerz300

Re: Limit mysql executions/ grabbing information from sites

Primefalcon

Re: Limit mysql executions/ grabbing information from sites

gamerz300

Re: Limit mysql executions/ grabbing information from sites

Confuser

Re: Limit mysql executions/ grabbing information from sites

gamerz300

Re: Limit mysql executions/ grabbing information from sites