UBB.Dev
Posted By: fishtails data retrieval - public domain - 11/11/2002 5:04 AM
I'm working on a script which pulls tide data from NOAA, parses it and the inserts into my PAL page. You can see it working at http://www.fish-tails.net

Anyway, I need a little help on regexp, and maybe some suggestions.

Ideally, I would like to pull the data from NOAA, strip html completely out, and then insert the data into mysql. Once the data is stored, I can then write a script so that the user can select the reference stations they want to be displayed on the home page, or in their My Home.

Currently, I have it loading directly to the home page without stripping html. Here is the code I'm using.

code:

$month = date("n");
$day = date("j");
$year = date("Y");
$next = (date("j")+1);
$next1 = $next+1;
$next2 = $next1+1;
$next3 = $next2+1;
$tide1 = "$month/$day/$year";
$tide2 = "$month/$next/$year";
$tide3 = "$month/$next1/$year";
$tide4 = "$month/$next2/$year";
$tide5 = "$month/$next3/$year";

//$url = 'http://www.co-ops.nos.noaa.gov/tides/seFB.html';
$url = 'http://co-ops.nos.noaa.gov/tides/get_pred.shtml?stn=0030+Fernandina+Beach&secstn=Simpson+Creek,
+A1A+highway+bridge&thh=+0&thm=05&tlh=+0&tlm=1
9&hh=*0.84&hl=*0.65';

$lines_array = file($url);

$lines_string = implode('', $lines_array);
eregi("$tide1(.*)$tide5", $lines_string, $head);
//eregi("<BODY BGCOLOR="#FFFFFF">(.*)</body>", $lines_string, $head);
//eregi("<pre>(.*)</pre>", $lines_string, $head);

//$head = eregi_replace("</tr><tr><td align="right" width="60">$tide5</td></tr>", "</tr>", $head);



echo "I'm running some tests now on the tides program I'm writing. Please use these with discretion. Thanks. DATA from NOAA";
echo "<br><b>Simpson Creek, A1A highway bridge 2002 Water Level Predictions</b>";
echo "<br>(Reference station - Fernandina Beach)";
echo "<br>Corrections - Time Diff. (High 0 hr. 05 min. Low 0 hr. 19 min.)";
echo "<br>Hgt. Diff. (High *0.84 Low *0.65)";
echo "<br>All times listed are in Local Time, and all heights are in Feet referenced to Mean Lower Low Water (MLLW).";
echo "<br><hr><table><tr>";
echo "<th width="60">Date</th><th width="60">Time</th><th></th><th width="40">Ht.</th><th>     
</th><th>Time</th><th></th><th width="40
">Ht.</th><th>     
</th><th>Time</th><th></th><th width="40
">Ht.</th><th>     
</th><th>Time</th><th></th><th width="40
">Ht.</th></tr><tr><td>";

echo "<font face='arial'>";


echo $head[0];


echo "</td></tr></table>";



You will note I have two $url with one commented out. The one that is commented is the one I think woul be the easiest to work with, but then I'm not sure, which is why I'm here.

The second one is the one that I'm using on my home page. Now, first off, I only want 4 days displayed, but if you look at my home page a new line with a fifth day starts. I've tried preg_replace and ereg_replace but to no avail. Anyone have a good understanding of regexp that can help me out?

Regarding the database. Looking for suggestions here. Should I set up a cron (I have an idea what it does but have never used it) to pull the data I want nightly, insert it into a db as one record, then have my pal box pull from that record? I would think that way there would be faster page load times. I'm open for suggestions here.

Any help is appreciated.
Posted By: smilesforu Re: data retrieval - public domain - 02/03/2003 2:36 PM
Did you make any headway on this? I would like to do something similar in my neck of the woods. (Washington)
Posted By: fishtails Re: data retrieval - public domain - 02/03/2003 11:33 PM
I'm still working on it. You can check out what I have so far here. http://www.fish-tails.net/threads/tidecal.php
Posted By: smilesforu Re: data retrieval - public domain - 02/04/2003 12:10 AM
Very nice.. Is this cached? or does it draw on their server every time?
Posted By: dimopoulos Re: data retrieval - public domain - 02/04/2003 12:14 AM
Nice!!!

Nikos
Posted By: fishtails Re: data retrieval - public domain - 02/04/2003 12:25 AM
Thanks. For the moment it draws on their server each request. I have to learn more regarding cache. Originally I wanted to get the data and insert into my db, but I haven't gotten that far yet. The best approach would be, I think, to cache it. Any help in this area is appreciated.
Posted By: smilesforu Re: data retrieval - public domain - 02/04/2003 12:50 PM
I use to use some commercial scripts that would strip and cache the sites I wanted. These worked great until they stopped maintaining them. It was in perl and the code was garbled, they used regex and formatting to get them squared away.
It is much nicer to see the open code so adjustments can be made. My php skill are beginner level still... was focused on learning the perl language but moving to php I think is a better option.
I hope to help in the future as I can get up to speed... Will kep my eyes open for some php cache scripts we can use on this sort of thing. Might be something over at the php project site.
Posted By: smilesforu Re: data retrieval - public domain - 02/07/2003 4:54 AM
I see the iip integration is using cache. Might be some useful to look at how its used in that script.
Posted By: smilesforu Re: data retrieval - public domain - 02/07/2003 8:57 AM
lol I tried putting your text to work... didn't fair so well

http://www.steelheader.net/tide/tidecal.php
Posted By: fishtails Re: data retrieval - public domain - 02/07/2003 7:37 PM
That code doesn't work. Try my Tide Sticker! Demo at http://www.fish-tails.net in one of the custom pal boxes.

Attached File
67943-tidesticker.txt  (299 downloads)
Posted By: smilesforu Re: data retrieval - public domain - 02/08/2003 3:09 AM
got it working!!! Nice job.
Posted By: smilesforu Re: data retrieval - public domain - 02/08/2003 7:43 AM
I ran into trouble putting this into a pal box. Got a parse error right where code begins.

Should I be including it some other way? A ssi?

Works perfect when I call it from a url. Sure its just me but thought I would check.
Thanks
Posted By: fishtails Re: data retrieval - public domain - 02/08/2003 7:20 PM
All you need to do is add this to your custom pal box.

include ("path/to/tidesticker.php");
Posted By: smilesforu Re: data retrieval - public domain - 02/09/2003 10:45 AM
Thanks will get to it tomorrow... fish call me
Posted By: fishtails Re: data retrieval - public domain - 02/09/2003 3:17 PM
I know what you mean! After all, we do have fishing sites that need reports. I fished out of a kayak for the first time the other day. A challenging experience.

[]http://www.fish-tails.net/photopost/data/3000/2img_0014-med.jpg[/]
Posted By: smilesforu Re: data retrieval - public domain - 02/10/2003 8:20 AM
Looks like a good time. I always feel like I am going to tip over in one of those.
Posted By: smilesforu Re: data retrieval - public domain - 02/12/2003 4:27 AM
Got it up and running with some additional tides showings.

http://www.steelheader.net/ubbthreads/index.php
Posted By: smilesforu Re: data retrieval - public domain - 02/12/2003 12:45 PM
Seems to be fairly slow loads since I added the extra tides.
Searched for a cache script we could use on it.
Benedetto Patrice code posted on webdev
Code
 // First create a folder named "cache"  where the files will be written <br />// define the fonction <br />function BZ_cache ($url,$cacheName) <br />{ <br />global $cache ; <br />global $QUERY_STRING ; <br />// You may use the naming convention of you choice here <br />$filename = "cache/".$cacheName.",".$QUERY_STRING.".html" ; <br /><br />// You may use test the date of the cache  here <br />    if ( file_exists (   $filename) ) <br />       {  <br />       readfile ($filename) ; <br />       return 1 ; ///  Ok i'v send the html page   <br />       } <br />else <br />      { <br />     if ( ! isset ($cache ) ) <br />        { <br />         $fcontents = join ('', file ($url."?".$QUERY_STRING."&cache=t")); <br />         $fp = fopen ($filename , "w"); <br />         fwrite ($fp, $fcontents  ); <br />         fclose ( $fp) ; <br />            return 0 ;   ///  i'v to execute the file <br />         } <br />      } <br />} <br />// then put this line at the TOP of you file ( before any include ) <br />//  if ( BZ_cache ("COMPLETE URL WITHOUT PARAMS" , "AN_IDENTIFIER" ) )   exit () ; <br /><br /><br />// Example in the file http://azerclic.labynet.org/doc.php3 <br />// the first line is <br />// if ( BZ_cache ("http://azerclic.labynet.org/doc.php3" , "doc" ) )   exit () ; <br />// you will see  the cached files in the  folder http://azerclic.labynet.org/cache/ <br />// That's all <br /> 
Posted By: smilesforu Re: data retrieval - public domain - 02/25/2003 12:28 PM
Just did a cache with a cron job on a tide script puller. Had to do a rewrite with help from one of my members to add other tides for the pal.

I have the script pulling the data witha regex and saving it to a text file.... then I use a include on the text file. Set the cron up to run the script so it keeps the text file updated. Loads like a dream... Thanks for the inspiration to get me over the hump on this. Using this same style of a feature to fetch the latest news and regulations, river levels.... next is weather.
Very close to having this setup for a mysql database, with cron for river levels. It will also have user login cookie so users can save there tides/river levels of choice.

BTW I used Fastcron.com for remote pull on the scripts since it was cheaper and very easy to setup. Cron job is the way to go!!

Not sure whether to include this with language files or not.. but probably will once its done.

Any day one a couple minor bugs are worked out...

Now back to your regular scheduled fishing trip []http://www.steelheader.net/ubbthreads/images/graemlins/jumping%20king.GIF[/]


Posted By: chillin Re: data retrieval - public domain - 03/01/2003 8:10 AM
very awesome guys, i think i'll hack this out for use with mountain/avalanche forecasts from NOAA as well, this will be a wonderful feature.
Posted By: chillin Re: data retrieval - public domain - 03/01/2003 11:10 PM
thanks for the inspiration, i got something similar working with mountain weather and avalanche forecasts from NOAA as well
Posted By: smilesforu Re: data retrieval - public domain - 11/04/2003 11:29 AM
Mine tide program crashed.. doesn't regex out the tide data from noaa properly so it returns a blank data field. Noaa must have changed something and my regex no longer works.
Posted By: smilesforu Re: data retrieval - public domain - 12/14/2003 4:19 AM
Mine is broke... not sure why it doesn't work but comes back with no tides. Anybody care to take a look see if it works on there server or if its just me.
Posted By: omegatron Re: data retrieval - public domain - 12/14/2003 6:27 AM
If you want I can try it out for you.
Posted By: smilesforu Re: data retrieval - public domain - 12/14/2003 6:47 AM
yes please
Posted By: fishtails Re: data retrieval - public domain - 12/25/2003 12:38 AM
They've changed the set up at NOAA. I'll try to get you the new code by this weekend.
Posted By: smilesforu Re: data retrieval - public domain - 12/25/2003 8:10 AM
thank you thank you ...
Posted By: omegatron Re: data retrieval - public domain - 12/25/2003 2:28 PM
And Fishtails ownes the 39.95 program LOL
Posted By: smilesforu Re: data retrieval - public domain - 12/26/2003 4:14 AM
I see the program now that you mentioned it. I don't think that is the cause of the trouble. Look forward to finding out.
Posted By: omegatron Re: data retrieval - public domain - 12/26/2003 5:28 AM
Well there seems to be a new cgi engine running and thus your data fields in the regex do not match and are altered thus your data is not pulled.
Posted By: fishtails Re: data retrieval - public domain - 12/27/2003 10:54 AM
Umm, that's only 9.95. Actually, you can see the tides in a calendar format here. The new year is coming up, so I gotta get it squared away in a couple of days. Not enough time in the day. sigh!
Posted By: smilesforu Re: data retrieval - public domain - 12/27/2003 12:33 PM
Nicely done.. looks like a well spent 10 bucks
Posted By: smilesforu Re: data retrieval - public domain - 01/12/2004 10:59 PM
Fishtails any headway on this one? Willing to pay for a working version so I can give this to my users and myself or couse
Posted By: fishtails Re: data retrieval - public domain - 01/13/2004 3:12 AM
Um, yeah, I forgot. It's done. I've got it running out of a database, i.e. first you extract the data from NOAA, insert it into a database. Then your script pulls it from the database so it's tons faster. No need for cron because the tide predictions are done from certain 'harmonic constants' , so basically no matter when you go to NOAA and get tidal predictions, they are the same each time you go because they are using the same 'hc's' each time. So, you can get the tides and insert them into a db one time during the year, and just check them occasionally to ensure that the data matches what you have with what NOAA has.

You can see the working version of the 'Tide Sticker' at http://www.fish-tails.net in the left hand column. It's not in a pal bax because I'm currently not using IIP for my home page due to some weird problems. However, it's just a matter of inserting the code into a cusompalbox. You can see it working in a pal box at http://www.fish-tails.net/index3.php (the one that was giving me problems and suddenly is working again). That's IIP 5.3.1, but it shouldn't matter which IIP you're using since it's just basic php code added to a a pal box.

I also have a calendar running at http://www.fish-tails.net/threads/tidecal.php , which has a drop down box to grab the station for where ever you want and displays it in calendar format. It pulls from the same tables as the Tide Sticker. Though it is working flawlessly, there is some code cleanup I want to do for the calendar.

I know you want the Tide Sticker. Send me a PM with an email and I'll send you the scripts as soon as I do some cleanup. There is one file for getting the data and inserting into a table. The other script is for the Tide Sticker itself. Once you've installed them and you fel like it's worth something to you, you can let me know and I'll give you my paypal address.

If you want the calendar as well, that can be dones as well, but as I stated, I need to do some definite code cleanup for that. Let me know if you want that one as well, but I need more time to clean it up.

Please keep in mind that I have many Florida fishing sites that would love to have my site go belly up, but they want my tides calendar and sticker really bad. Plus, they're trying to figure out how I've got weather (integrated with threads) on my site as well (members only feature). So, with that being said, you must agree NOT to divulge any of the code for either the tides calendar or the sticker to anyone. I am hoping that one of the big players (no names but they have some big fishing publications out, as well as big fishing sites) might see my scripts valuable, which would equate to a big "Fish ON!" inangling terms.

Anyway, send me an email in the PM. I'll get it to you soon as I can.

Tight Lines and either fish or cut bait!
Posted By: smilesforu Re: data retrieval - public domain - 01/13/2004 11:35 AM
Thanks I sent you a PM with my email.
Posted By: fishtails Re: data retrieval - public domain - 01/13/2004 5:46 PM
Check your email Marty. Enjoy.
Posted By: smilesforu Re: data retrieval - public domain - 01/13/2004 11:24 PM
Never made it through.. checked my spam filter. Please send to my other email. Sent you a new private email.
Thanks
Posted By: YakFool Re: data retrieval - public domain - 12/22/2004 5:31 AM
Any chance I could get a copy of tidecal.php and the scripts? It's awesome.
© UBB.Developers