CNV.com Message Forums Forum owned by CNV.com Inc.
Wholesale and Drop Ship Adult Toys at SextoyClub.com
Sex Toys Affiliate program at AdultShopping.com
 

Go Back   CNV.com Message Forums > Wholesale and Drop Ship: SexToyClub.com > Datafeeds

Reply
 
Thread Tools Display Modes
  #1  
Old 05-22-2015, 03:59 AM
SxcDev SxcDev is offline
Newbie
 
Join Date: Mar 2014
Posts: 15
Default Issues with discontinued product datafeed

For the discontinued.csv datafeed, there are 2 columns, the first being the part number, and the second being a date. What does that date represent?

The date field seems to contain invalid values, such as:

"CNVELD-XP2020","1969-12-31" <= a representation of a 0 value for a date?
"CNVELD-XP2070","" <= non-existent date
"CNVELD-XRAD112","0000-00-00" <= invalid date

Parsing that entire CSV file is really inefficient and extremely time consuming because of the processing required to check discontinued products against current items in the store.

If the date field represents when the product was discontinued, then great, since we can at least search for the entries with the most recent dates since the datafeed was last parsed, and only process those specific newly discontinued items. The only concern here is that future entries not containing an invalid value in the date field, which makes it completely useless.

So, what does that date represent, and can you guarantee that newly added items to that feed will have a valid date?
Reply With Quote
  #2  
Old 05-22-2015, 10:32 AM
CNVAdmin CNVAdmin is offline
Administrator
 
Join Date: Dec 2010
Location: Viva Las Vegas
Posts: 382
Default

Hello SxcDev,

As you have seen the dates are unreliable in many cases, this information comes from the vendors who get it from the manufacturers. Sometimes the information is accurate and other times it is not.

Basically the date information can be ignored, because what counts is the SKU / Product Model Number for all three stock information data feeds. If the SKU is in any of those data feeds (InStock, OutofStock or discontinued) you can be sure that they belong in that feed when it was created, which happens to be every 15 minutes for the stock information data feeds.

Quote:
Parsing that entire CSV file is really inefficient and extremely time consuming because of the processing required to check discontinued products against current items in the store.
This statement is inaccurate. Downloading / adding to a table / and updating any shopping cart with the discontinued and the OutofStock data feeds takes all of 5 to 10 seconds for each feed. Expending 20 seconds every 20 to 40 minutes is not my definition of "extremely time consuming".

If you can not get this done within 5 to 10 seconds, then you need to check your script for efficiency and also ensure the shopping cart data base is properly indexed for the areas you are reading / updating.

I've found the fastest and most efficient way to accomplish this task is:
  1. First check if the remote file is newer than the local file.

  2. Also check if the remote file is a different size then the local file.

  3. If one of the above is true, or it's been over 90 minutes since the last download, then move on to step 4. Otherwise stop the script.

  4. Copy the remote data feed and replace the local file, use "Load Data Local Infile" to put the data feed into it's own table. This can be done in 3 to 6 seconds for both of these tasks depending on current Internet Latency.

  5. Select each product model number in the shopping carts products table, check if that model number is present in the data feed table.

  6. If it is present, adjust your shopping cart accordingly eg. mark as out of stock / 0 quantity or delete / mark inactive if discontinued. Steps 5 and 6 constitute the remaining seconds to complete the whole shopping cart database.
__________________
--
Best Regards,
Mike
CNV.com
Reply With Quote
  #3  
Old 05-22-2015, 06:33 PM
SxcDev SxcDev is offline
Newbie
 
Join Date: Mar 2014
Posts: 15
Default

Hi Mike, thank you for the information regarding the date field. It's a shame that it isn't being used, since it would have helped a lot.

Quote:
Originally Posted by Mike View Post
This statement is inaccurate. Downloading / adding to a table / and updating any shopping cart with the discontinued and the OutofStock data feeds takes all of 5 to 10 seconds for each feed. Expending 20 seconds every 20 to 40 minutes is not my definition of "extremely time consuming".

If you can not get this done within 5 to 10 seconds, then you need to check your script for efficiency and also ensure the shopping cart data base is properly indexed for the areas you are reading / updating.
I should have been more specific with my statement regarding it being processing intensive. It has nothing to do with the process of downloading the datafeeds, nor the frequency or steps required for processing them. It is specifically to do with WooCommerce/Wordpress.

A lot goes on behind the scene when searching for an SKU via the WP API, which is what slows down the entire process. A simple loop for approximately 170000 entries in the discontinued datafeed that compares them to the existing SKUs in the store took over 8 minutes. You can see how this is completely unreasonable, and while processing it slowed down the response time of the entire site. The code to do this is literally a simple foreach loop and WP query to search for the SKU in the store, nothing more than that.

Anyhow, it looks like I may have to try another approach, perhaps by hitting the WP database tables directly (which isn't recommended), or using custom data tables synced with the product entries in the main store tables to handle this look-up process.

Thanks again for the info.
Reply With Quote
  #4  
Old 05-22-2015, 10:03 PM
CNVAdmin CNVAdmin is offline
Administrator
 
Join Date: Dec 2010
Location: Viva Las Vegas
Posts: 382
Default

Hello J,

I see what you're saying now. I've tried in the past to make custom software for Woo / WP, but in the end gave it up because it would require too many alterations to the database structure and indexing system, which could make any future upgrade in Woo / WP problematic.

Even though Woo does a good job of turning WP into a shopping cart, in the end the data base was built for being a blog and not an ecommerce platform. You know the old saying, a wolf in sheep's clothing is still a wolf.

To accomplish what you want, I don't see any other alternative but to deal directly with the database and avoid the WP API. If you only do this with the stock data feeds update you could make a simple script that only does this one thing and do it more efficiently than the WP API.
__________________
--
Best Regards,
Mike
CNV.com
Reply With Quote
  #5  
Old 05-22-2015, 10:46 PM
SxcDev SxcDev is offline
Newbie
 
Join Date: Mar 2014
Posts: 15
Default

Quote:
Originally Posted by Mike View Post
Even though Woo does a good job of turning WP into a shopping cart, in the end the data base was built for being a blog and not an ecommerce platform. You know the old saying, a wolf in sheep's clothing is still a wolf.
That is precisely the problem. I've got a product WooCommerce importer working for the main product data feed, but it's amazingly slow. I was wondering what I was doing wrong, so I tested out other WP importers, and they all have exactly the same issue, slow importing speed. It's just the core WP code that is getting in the way in order to do what should be straightforward tasks, like adding and updating products.

Oh well, it's an interesting challenge.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 12:05 PM.


Powered by vBulletin® Version 3.8.1
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright 2008-2010 CNV.com