Tuesday, December 10, 2013

The Issues of Migrating from Bing Ads API V8 to V9

When I upgraded from Bing Ads API V8 to V9, a couple of issues popped up, and I tried out following workarounds to resolve them, please check out.
1. Partial Success for Ads and Keywords: When adding, updating, or deleting ads or keywords in batches of one or more, the operation may succeed for some and fail for part of the batch.
Here I tried to add 4 keywords, 2 of them were invalid. So the response has keyword ids as below, that’s corrent, 2 of them should not have id. But the problem happened when Axis tried to convert this id list to long[].  Please see the exception below. any idea to fix this issue?

                  <KeywordIds xmlns:a="http://schemas.datacontract.org/2004/07/System" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
                        <a:long>15126706305</a:long>
                        <a:long i:nil="true" />
                        <a:long>15126706306</a:long>
                        <a:long i:nil="true" />
                  </KeywordIds>

ERROR [pool-8-thread-1] (BeanPropertyTarget.java:135) - Could not convert org.apache.axis.encoding.ser.ArrayDeserializer$ArrayListExtension to bean field 'keywordIds', type [J
ERROR [pool-8-thread-1] (Call.java:2469) - Exception:
java.lang.IllegalArgumentException
      at org.apache.axis.encoding.ser.BeanPropertyTarget.set(BeanPropertyTarget.java:157)
      at org.apache.axis.encoding.DeserializerImpl.valueComplete(DeserializerImpl.java:249)

Changing from long[] to Long[] will fix this problem

2. Bulk download campaign data or performance data potentially has extra Error row for each of  BulkDownloadEntity as documented here. which means the first column Type should precisely has following set of values, looks ugly! I think the error should be combined to it's above entity row.

<xs:simpleType name="BulkDownloadEntity">
  <xs:list>
    <xs:simpleType>
      <xs:restriction base="xs:string">
        <xs:enumeration value="Campaigns" />
        <xs:enumeration value="AdGroups" />
        <xs:enumeration value="Ads" />
        <xs:enumeration value="Keywords" />
        <xs:enumeration value="CampaignNegativeKeywords" />
        <xs:enumeration value="AdGroupNegativeKeywords" />
        <xs:enumeration value="CampaignTargets" />
        <xs:enumeration value="AdGroupTargets" />
        <xs:enumeration value="CampaignNegativeSites" />
        <xs:enumeration value="AdGroupNegativeSites" />
        <xs:enumeration value="CampaignSiteLinksAdExtensions" />
        <xs:enumeration value="CampaignProductAdExtensions" />
        <xs:enumeration value="CampaignLocationAdExtensions" />
        <xs:enumeration value="CampaignCallAdExtensions" />
        <xs:enumeration value="AdGroupProductTargets" />
        <xs:enumeration value="AdGroupSiteLinksAdExtensions" />
        <xs:enumeration value="CampaignsError" />
        <xs:enumeration value="AdGroupsError" />
        <xs:enumeration value="AdsError" />
        <xs:enumeration value="KeywordsError" />
        <xs:enumeration value="CampaignNegativeKeywordsError" />
        <xs:enumeration value="AdGroupNegativeKeywordsError" />
        <xs:enumeration value="CampaignTargetsError" />
        <xs:enumeration value="AdGroupTargetsError" />
        <xs:enumeration value="CampaignNegativeSitesError" />
        <xs:enumeration value="AdGroupNegativeSitesError" />
        <xs:enumeration value="CampaignSiteLinksAdExtensionsError" />
        <xs:enumeration value="CampaignProductAdExtensionsError" />
        <xs:enumeration value="CampaignLocationAdExtensionsError" />
        <xs:enumeration value="CampaignCallAdExtensionsError" />
        <xs:enumeration value="AdGroupProductTargetsError" />
        <xs:enumeration value="AdGroupSiteLinksAdExtensionsError" />
      </xs:restriction>
    </xs:simpleType>
  </xs:list>
</xs:simpleType>

So the campaign data parser has to deal with the potential Error rows, take the Keyword for example:
 @Override
 public List<BiddableKeyword> getBiddableKeywords(int batchSize) throws IOException {
  List<BiddableKeyword> keywords = new ArrayList<BiddableKeyword>();
  if (batchSize > 0 && !passedTypes.contains(TYPE_KEYWORD)) {
   boolean foundType = false;
   BiddableKeyword keyword = null;
   while (currentRow != null) {
    String rowType = getRowType(currentRow);
    if (rowType.equals(TYPE_KEYWORD)) {
     foundType = true;
     if (keywords.size() == batchSize)
      break; // check size first
     keyword = getBiddableKeyword(currentRow);
     keywords.add(keyword);
    } else if (rowType.equals(TYPE_KEYWORD + ERROR_SUFFIX)) {
     // handle the separate keyword error row!!!
     Long keywordApiId = Long.valueOf(getCellValue(currentRow, COLUMN_ID));
     if (keyword.getApiId().equals(keywordApiId)) {
      keyword.setDisapprovalReasons("Reason code: " + getCellValue(currentRow, COLUMN_EDITORIAL_REASON_CODE));
     }
    } else if (foundType) {
     passedTypes.add(TYPE_KEYWORD);
     break;
    }
    currentRow = reader.readNext();
   }
  }
  return keywords;
 }

 private BiddableKeyword getBiddableKeyword(String[] dataRow) {
  BiddableKeyword keyword = new BiddableKeyword();
  keyword.setSearchEngine(SearchEngine.BING);
  keyword.setUserStatus(getCellValue(dataRow, COLUMN_STATUS).toUpperCase());
  keyword.setApiId(getCellValue(dataRow, COLUMN_ID));
  keyword.getAdGroup(true).setApiId(getCellValue(dataRow, COLUMN_PARENT_ID));
  keyword.getAdGroup().getCampaign(true).setCampaignName(getCellValue(dataRow, COLUMN_CAMPAIGN));
  keyword.getAdGroup().setAdGroupName(getCellValue(dataRow, COLUMN_AD_GROUP));
  keyword.setKeywordText(getCellValue(dataRow, COLUMN_KEYWORD));
  if (keyword.getUserStatus() != UserStatus.DELETED) {
   String matchType = getCellValue(dataRow, COLUMN_MATCH_TYPE);
   keyword.setMatchType(matchType == null ? null : matchType.toUpperCase());
   keyword.setMaxCpc(getCellValue(dataRow, COLUMN_BID));
   keyword.setDestinationUrl(getCellValue(dataRow, COLUMN_DESTINATION_URL));
   keyword.setAdParam1(getCellValue(dataRow, COLUMN_PARAM1));
   keyword.setAdParam2(getCellValue(dataRow, COLUMN_PARAM2));
   keyword.setAdParam3(getCellValue(dataRow, COLUMN_PARAM3));
   keyword.setApprovalStatus(getCellValue(dataRow, COLUMN_EDITORIAL_STATUS));
   keyword.setDisapprovalReasons(getCellValue(dataRow, COLUMN_EDITORIAL_REASON_CODE));
   keyword.setQualityScore(getCellValue(dataRow, COLUMN_QUALITY_SCORE));
   keyword.identifyBMM(); // identify BMM match type!
  }
  return keyword;
 }

3. One more issue I found is also for the bulk download, when you deleted some negative keywords, the downloaded report will show these keywords as  Deleted, BUT  the Keyword column is empty, and I don’t know which one was deleted since the negative keyword has no Id, this is not helpful for the merge,

So to merge negative keywords, I have to treat them as a whole set, means you get all negative keywords for some campaigns or ad groups, and fully merge to db. the parser to get negative keywords looks like this:

 @Override
 public List<NegativeKeyword> getCampaignNegativeKeywords(int batchSize) throws IOException {
  List<NegativeKeyword> negativeKeywords = new ArrayList<NegativeKeyword>();
  if (batchSize > 0 && !passedTypes.contains(TYPE_CAMPAIGN_NEGATIVE_KEYWORD)) {
   boolean foundType = false;
   while (currentRow != null) {
    String rowType = getRowType(currentRow);
    if (rowType.equals(TYPE_CAMPAIGN_NEGATIVE_KEYWORD)) {
     foundType = true;
     if (negativeKeywords.size() >= batchSize)
      break; // check size first!
     long campaignId = Long.valueOf(getCellValue(currentRow, COLUMN_PARENT_ID));
     negativeKeywords.addAll(getCampaignNegativeKeywords(campaignId));
     continue;
    } else if (rowType.equals(TYPE_CAMPAIGN_NEGATIVE_KEYWORD + ERROR_SUFFIX)) {
     // skip error line!     
    } else if (foundType) {
     passedTypes.add(TYPE_CAMPAIGN_NEGATIVE_KEYWORD);
     break;
    }
    currentRow = reader.readNext();
   }
  }
  return negativeKeywords;
 }

 // just for this single campaign!
 private List<NegativeKeyword> getCampaignNegativeKeywords(long campaignId) throws IOException {
  List<NegativeKeyword> negativeKeywords = new ArrayList<NegativeKeyword>();
  if (campaignId > 0) {
   while (currentRow != null) {
    String rowType = getRowType(currentRow);
    if (rowType.equals(TYPE_CAMPAIGN_NEGATIVE_KEYWORD)) {
     NegativeKeyword keyword = getCampaignNegativeKeyword(currentRow);
     if (campaignId == keyword.getAdGroup().getCampaign().getApiId()) {
      if (!(keyword.getUserStatus() == UserStatus.DELETED && keyword.getKeywordText() == null)) // The delete one doesn't have keyword text, NOT helpful for merge!
       negativeKeywords.add(keyword);
     } else {
      break;
     }
    } else if (rowType.equals(TYPE_CAMPAIGN_NEGATIVE_KEYWORD + ERROR_SUFFIX)) {
     // skip error line!
    } else {
     break;
    }
    currentRow = reader.readNext();
   }
  }
  return negativeKeywords;
 }

 private NegativeKeyword getCampaignNegativeKeyword(String[] dataRow) {
  NegativeKeyword negativeKeyword = new NegativeKeyword();
  negativeKeyword.setMapLevel(MapLevel.CAMPAIGN);
  negativeKeyword.setUserStatus(getCellValue(dataRow, COLUMN_STATUS).toUpperCase());
  negativeKeyword.getAdGroup(true).getCampaign(true).setApiId(getCellValue(dataRow, COLUMN_PARENT_ID));
  negativeKeyword.getAdGroup().getCampaign().setCampaignName(getCellValue(dataRow, COLUMN_CAMPAIGN));
  negativeKeyword.setKeywordName(SearchEngine.BING, getCellValue(dataRow, COLUMN_KEYWORD));
  return negativeKeyword;
 }


Handle Bing Ads API Errors or Exceptions

Last chapter, we talked about Handle Google AdWords API Errors or Exceptions . With the new Bing Ads API Version 9, The partial failure/success feature was introduced to the following operations, The PartialErrors element represents an array of BatchError objects that contain details for any entities that were not successfully added, updated, or deleted.
Take the AddKeywords operation for example, I am also trying to build a centralized the way to handle all kinds of errors or exceptions populated by Bing or the application itself.

 @Override
 public boolean addKeywords(Long adGroupId, List<BiddableKeyword> adGroupKeywords) {
  logger.debug("Adding {} keywords of ad group {}", adGroupKeywords.size(), adGroupId);
  boolean isAllGood = true;
  AddKeywordsRequest request = new AddKeywordsRequest();
  request.setAdGroupId(adGroupId);
  com.microsoft.bingads.CampaignManagement.v9.Keyword[] keywords = new com.microsoft.bingads.CampaignManagement.v9.Keyword[adGroupKeywords
    .size()];
  for (int i = 0; i < adGroupKeywords.size(); i++) {
   BiddableKeyword keyword = adGroupKeywords.get(i);
   keywords[i] = new com.microsoft.bingads.CampaignManagement.v9.Keyword();
   //keywords[i].setId(keyword.getApiId());
   keywords[i].setText(keyword.getKeywordText());
   keywords[i].setMatchType(getMatchType(keyword));
   if (keyword.getUserStatus() != null)
    keywords[i].setStatus(getUserStatus(keyword));
   if (keyword.getMaxCpc() != null)
    keywords[i].setBid(new Bid(keyword.getMaxCpc() == 0.0 ? null : keyword.getMaxCpc()));
   if (keyword.getDestinationUrl() != null)
    keywords[i].setDestinationUrl(keyword.getDestinationUrl());
   if (keyword.getAdParam1() != null)
    keywords[i].setParam1(keyword.getAdParam1());
   if (keyword.getAdParam2() != null)
    keywords[i].setParam2(keyword.getAdParam2());
   if (keyword.getAdParam3() != null)
    keywords[i].setParam2(keyword.getAdParam3());

  }
  request.setKeywords(keywords);
  // submit changes now!
  int triedTimes = 0;
  while (triedTimes < Constants.API_CALL_TRY_TIMES) {
   try {
    AddKeywordsResponse response = serviceStub.addKeywords(request);
    populateBatchErrors(adGroupKeywords, response.getPartialErrors());
    assignedReturnedKeywordIds(adGroupKeywords, response);
    isAllGood = response != null && isAllGood;
    return isAllGood;
   } catch (RemoteException exception) {
    triedTimes++;
    if (handleApiCallException(adGroupKeywords, exception, triedTimes))
     break;
   }
  }
  return isAllGood;
 }
Bing Ads V9 put the BatchError and ApiFaultDetail in different packages instead of a unified error package which makes thing awkward. so you have to define the populateBatchErrors() for each BatchError type and handleApiFaultDetail() for each ApiFaultDetail:

com.microsoft.bingads.CampaignManagement.v9.BatchError
com.microsoft.bingads.Reporting.v9.BatchError
com.microsoft.bingads.Optimizer.v9.BatchError
com.microsoft.bingads.AdIntelligence.v9.BatchError
 protected void populateBatchErrors(List<? extends ApiEntity> apiEntities, com.microsoft.bingads.CampaignManagement.v9.BatchError[] batchErrors) {
  for (com.microsoft.bingads.CampaignManagement.v9.BatchError batchError : batchErrors) {
   ApiEntity apiEntity = apiEntities.get(batchError.getIndex());
   if (apiEntity.getApiError() == null) {
    apiEntity.setApiError(new ApiError());
    ApiError apiError = apiEntity.getApiError();
    apiError.setItemIndex(batchError.getIndex());
    apiError.appendErrorString(batchError.getMessage());
    apiError.appendErrorType(batchError.getErrorCode());
    apiError.appendErrorString(batchError.getDetails());
   }
  }
 }
 // return true to break the loop
 protected boolean handleApiCallException(List<? extends ApiEntity> apiEntities, Exception exception, int triedTimes) {
  logger.debug("Got {} exception, tried {} times.", exception.getClass().getSimpleName(), triedTimes);
  if (exception instanceof EditorialApiFaultDetail) {
   // not neccessary to retry
   handleEditorialApiFaultDetail(apiEntities, (EditorialApiFaultDetail) exception);
   return true; // not neccessary to retry!
  } else if (exception instanceof AdApiFaultDetail) {
   handleAdApiFaultDetail(apiEntities, (AdApiFaultDetail) exception);
   return true; // not neccessary to retry!
  } else if (exception instanceof com.microsoft.bingads.CampaignManagement.v9.ApiFaultDetail) {
   handleApiFaultDetail(apiEntities, (com.microsoft.bingads.CampaignManagement.v9.ApiFaultDetail) exception);
   return true; // not neccessary to retry!
  } else if (exception instanceof com.microsoft.bingads.Reporting.v9.ApiFaultDetail) {
   handleApiFaultDetail(apiEntities, (com.microsoft.bingads.Reporting.v9.ApiFaultDetail) exception);
   return true; // not neccessary to retry!
  } else if (exception instanceof com.microsoft.bingads.Optimizer.v9.ApiFaultDetail) {
   handleApiFaultDetail(apiEntities, (com.microsoft.bingads.Optimizer.v9.ApiFaultDetail) exception);
   return true; // not neccessary to retry!
  } else if (exception instanceof com.microsoft.bingads.AdIntelligence.v9.ApiFaultDetail) {
   handleApiFaultDetail(apiEntities, (com.microsoft.bingads.AdIntelligence.v9.ApiFaultDetail) exception);
   return true; // not neccessary to retry!   
  } else {
   if (triedTimes == Constants.API_CALL_TRY_TIMES)
    handleDefaultException(apiEntities, exception);
  }
  return false;
 }

 protected void handleEditorialApiFaultDetail(List<? extends ApiEntity> apiEntities, EditorialApiFaultDetail faultDetail) {
  for (EditorialError editorialError : faultDetail.getEditorialErrors()) {
   ApiEntity apiEntity = apiEntities.get(editorialError.getIndex());
   // Seems the same index might occur more than once due to the different publisher countries, so only use the first one.
   if (apiEntity.getApiError() == null) {
    apiEntity.setApiError(new ApiError());
    ApiError apiError = apiEntity.getApiError();
    apiError.setItemIndex(editorialError.getIndex());
    apiError.appendErrorString(editorialError.getMessage());
    apiError.appendErrorType(editorialError.getErrorCode());
    if (editorialError.getDisapprovedText() != null && editorialError.getDisapprovedText().length() > 0) {
     apiError.appendErrorString(" [" + editorialError.getDisapprovedText() + "]");
    }
   }
  }
  for (com.microsoft.bingads.CampaignManagement.v9.OperationError operationError : faultDetail.getOperationErrors()) {
   for (ApiEntity apiEntity : apiEntities) {
    if (apiEntity.getApiError() == null)
     apiEntity.setApiError(new ApiError());
    ApiError apiError = apiEntity.getApiError();
    apiError.setItemIndex(ApiError.ALL_FAILED_INDEX);
    apiError.appendErrorString(operationError.getMessage());
    apiError.appendErrorType(operationError.getErrorCode());
    apiError.appendErrorString(operationError.getDetails());
   }
  }
  populateBatchErrors(apiEntities, faultDetail.getBatchErrors());
 }

 protected void handleAdApiFaultDetail(List<? extends ApiEntity> apiEntities, AdApiFaultDetail faultDetail) {
  for (ApiEntity apiEntity : apiEntities) {
   if (apiEntity.getApiError() == null)
    apiEntity.setApiError(new ApiError());
   ApiError apiError = apiEntity.getApiError();
   for (AdApiError error : faultDetail.getErrors()) {
    apiError.appendErrorString(error.getMessage());
    apiError.appendErrorType(error.getErrorCode());
    apiError.appendErrorString(error.getDetail());
   }
  }
 }

Monday, December 9, 2013

Handle Google AdWords API Errors or Exceptions

Both Google AdWords API and Bing Ads API defined a bunch of errors, most of these errors are not recoverable, you just need to check the error reason, fix your input data accordingly and send new operation request again. But some of the errors suggest to retry the same request in 30 seconds, for instance, CONCURRENT_MODIFICATION and RATE_EXCEEDED. Things become complicated when you enabled the partial failure feature due to you might get errors for partial change items or exceptions for the entire operation request. Here is a sample I am trying to centralize the handling of all possible errors or exceptions, so you don't have to duplicate the code here and there.

Let's say you have a method to add keywords as below:

 @Override
 public boolean addKeywords(List<BiddableKeyword> keywords) {
  logger.debug("Adding {} keywords...", keywords.size());
  List<AdGroupCriterionOperation> operations = new ArrayList<AdGroupCriterionOperation>();
  for (BiddableKeyword keyword : keywords) {
   // create keyword
   com.google.api.ads.adwords.axis.v201309.cm.Keyword newKeyword = new com.google.api.ads.adwords.axis.v201309.cm.Keyword();
   newKeyword.setText(keyword.getKeywordText());
   newKeyword.setMatchType(KeywordMatchType.fromValue(keyword.getApiMatchTypeValue()));
   // create ad group criterion
   BiddableAdGroupCriterion adGroupCriterion = new BiddableAdGroupCriterion();
   adGroupCriterion.setAdGroupId(keyword.getAdGroup().getApiId());
   adGroupCriterion.setCriterion(newKeyword);
   // update keyword status
   if (keyword.getUserStatus() != null) {
    UserStatus userStatus = UserStatus.fromValue(keyword.getUserStatusValue());
    adGroupCriterion.setUserStatus(userStatus);
   }
   // update destination url
   if (keyword.getDestinationUrl() != null) {
    adGroupCriterion.setDestinationUrl(keyword.getDestinationUrl());
   }
   // update bid amount/max cpc
   adGroupCriterion.setBiddingStrategyConfiguration(getBiddingStrategyConfiguration(keyword));
   // create operation
   AdGroupCriterionOperation operation = new AdGroupCriterionOperation();
   operation.setOperand(adGroupCriterion);
   operation.setOperator(Operator.ADD);
   // add operation
   operations.add(operation);
  }
  // submit changes now!
  int triedTimes = 0;
  while (triedTimes < Constants.API_CALL_TRY_TIMES) { // maximum 3 times
   try {
    AdGroupCriterionServiceInterface service = getAdWordsService(AdGroupCriterionServiceInterface.class);
    AdGroupCriterionReturnValue response = service.mutate(operations.toArray(new AdGroupCriterionOperation[0]));
    populateApiErrors(keywords, response.getPartialFailureErrors());
    assignReturnedCriterionIds(keywords, response);
    return response != null && updateAdParams(keywords);
   } catch (Exception exception) {
    triedTimes++;
    if (handleApiCallException(keywords, exception, triedTimes))
     break;
   }
  }
  return false;
 }

The maximum triedTimes is 3, populateApiErrors and handleApiCallException are the centralized methods to handle errors from both partial failures or thrown exceptions, the triedTimes to be passed into handleApiCallException which decides whether to abort or retry. Please check out following details:
 protected void populateApiErrors(List<? extends ApiEntity> apiEntities, com.google.api.ads.adwords.axis.v201309.cm.ApiError[] apiErrors) {
  if (apiErrors != null) {
   for (com.google.api.ads.adwords.axis.v201309.cm.ApiError apiError : apiErrors) {
    Matcher matcher = operationIndexPattern.matcher(apiError.getFieldPath());
    if (matcher.matches()) {
     int operationIndex = Integer.parseInt(matcher.group(1));
     if (apiEntities.size() > operationIndex) { // fix a potential issue!!!
      ApiEntity apiEntity = apiEntities.get(operationIndex);
      apiEntity.getApiError(true).setItemIndex(operationIndex);
      apiEntity.getApiError(true).setErrorType(apiError.getApiErrorType());
      apiEntity.getApiError(true).appendErrorString(apiError.getErrorString());
     }
    } else {
     for (ApiEntity apiEntity : apiEntities) {
      apiEntity.getApiError(true).setItemIndex(ApiError.ALL_FAILED_INDEX);
      apiEntity.getApiError(true).setErrorType(apiError.getApiErrorType());
      apiEntity.getApiError(true).appendErrorString(apiError.getErrorString());
     }
    }
   }
  }
 }

 protected boolean handleApiCallException(List<? extends ApiEntity> apiEntities, Exception exception, int triedTimes) {
  logger.debug("Got {} exception, tried {} times.", exception.getClass().getSimpleName(), triedTimes);
  if (exception instanceof ApiException) {
   ApiException apiException = (ApiException) exception;
   // Try everything possible to recover from the error
   for (com.google.api.ads.adwords.axis.v201309.cm.ApiError apiError : apiException.getErrors()) {
    if (apiError instanceof AuthenticationError)
     handleAuthenticationError((AuthenticationError) apiError);
    else if (apiError instanceof RateExceededError)
     handleRateExceededError((RateExceededError) apiError);
    else if (apiError instanceof DatabaseError)
     handleDatabaseError((DatabaseError) apiError);
    else if (apiError instanceof CustomerSyncError)
     triedTimes = handleCustomerSyncError((CustomerSyncError) apiError);
    else if (apiError instanceof InternalApiError)
     triedTimes = Constants.API_CALL_TRY_TIMES;
   }
   // Still failed, need to populate the errors
   if (triedTimes == Constants.API_CALL_TRY_TIMES) {
    populateApiErrors(apiEntities, apiException.getErrors());
    return true; // no need to try again!
   }
  } else if (exception instanceof NullPointerException) {
   populateDefaultException(apiEntities, exception);
   return true; // no need to try again!
  } else {
   if (triedTimes == Constants.API_CALL_TRY_TIMES)
    populateDefaultException(apiEntities, exception);
  }
  return false;
 }

 protected void handleAuthenticationError(AuthenticationError apiError) {
  if (apiError.getReason() == AuthenticationErrorReason.GOOGLE_ACCOUNT_COOKIE_INVALID) {
   try {
    adWordsSession.getOAuth2Credential().refreshToken();
   } catch (Exception e) {
    // do nothing
   }
  } else {
   logger.error("Service call failed for authentication reason: " + apiError.getReason());
  }
 }

 protected void handleRateExceededError(RateExceededError error) {
  logger.warn("A api call failed due to rate exceeded, will retry after {} seconds", Constants.API_CALL_WAIT_INTERVAL / 1000);
  waitForNextAPICall();
 }

 protected void handleDatabaseError(DatabaseError error) {
  if (error.getReason() == DatabaseErrorReason.CONCURRENT_MODIFICATION) {
   logger.warn("A api call failed due to concurrent modification, will retry after {} seconds", Constants.API_CALL_WAIT_INTERVAL / 1000);
   waitForNextAPICall();
  }
 }

 protected int handleCustomerSyncError(CustomerSyncError error) {
  return Constants.API_CALL_TRY_TIMES;
 }

Use the WSDL2Java to build Bing Ads API proxy library

I worked out a Windows batch script to generate and build the Bing Ads API client library conveniently, please check it out.

For Bing Ads API V8:

wsdl2java-v8.bat
@echo off
echo Notes:
echo Download and unzip axis-bin-1_4.zip, put this wsdl2java.bat to axis-1_4\lib and run it.
echo A jar file adcenter-v8.jar would be generated if all good, please use it in your app.
echo mail.jar and activiation.jar are optional, not required here.

echo -----------------------
echo Converting Ad Intelligence Service...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://adcenterapi.microsoft.com/Api/Advertiser/v8/CampaignManagement/AdIntelligenceService.svc?wsdl

echo Converting Bulk Service...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://adcenterapi.microsoft.com/Api/Advertiser/v8/CampaignManagement/BulkService.svc?wsdl 

echo Converting Campaign Management Service...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://adcenterapi.microsoft.com/Api/Advertiser/v8/CampaignManagement/CampaignManagementService.svc?wsdl

echo Converting Customer Billing Service...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://sharedservices.adcenterapi.microsoft.com/Api/Billing/v8/CustomerBillingService.svc?wsdl 

echo Converting Customer Management...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://sharedservices.adcenterapi.microsoft.com/Api/CustomerManagement/v8/CustomerManagementService.svc?wsdl

echo Converting Notification Service...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://sharedservices.adcenterapi.microsoft.com/Api/Notification/v8/NotificationService.svc?wsdl

echo Converting Optimizer Service...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://adcenterapi.microsoft.com/Api/Advertiser/v8/Optimizer/OptimizerService.svc?wsdl 

echo Converting Reporting Service...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://adcenterapi.microsoft.com/Api/Advertiser/v8/Reporting/ReportingService.svc?wsdl

echo Compiling java source code...
javac -cp .;* -encoding UTF-8 com/microsoft/adapi/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/adcenter/v8/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/adcenter/api/customerbilling/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/adcenter/api/customermanagement/Entities/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/adcenter/api/customermanagement/Exception/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/adcenter/api/customermanagement/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/adcenter/api/notifications/Entities/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/adcenter/api/notifications/*.java
javac -cp .;* -encoding UTF-8 org/datacontract/schemas/_2004/_07/Microsoft_AdCenter_Advertiser_CampaignManagement_Api_DataContracts/*.java
javac -cp .;* -encoding UTF-8 org/datacontract/schemas/_2004/_07/Microsoft_AdCenter_Advertiser_Optimizer_Api_DataContracts/*.java
javac -cp .;* -encoding UTF-8 org/datacontract/schemas/_2004/_07/Microsoft_AdCenter_Advertiser_Optimizer_Api_DataContracts_Entities/*.java
javac -cp .;* -encoding UTF-8 org/datacontract/schemas/_2004/_07/Microsoft_AdCenter_Shared_Api/*.java

echo Creating jar file adcenter-v8.jar...
jar cf adcenter-v8.jar com/microsoft/* org/datacontract/*

echo WSDL2Java V8 Converting completed!

echo Use following command to add to your maven repository
echo mvn install:install-file -Dfile=adcenter-v8.jar -DgroupId=com.microsoft -DartifactId=adcenter -Dversion=v8 -Dpackaging=jar
For Bing Ads API V9, We need 2 steps as we have to fix something before compiling:

wsdl2java-v9-step1.bat
@echo off
echo Running The Step 1......
echo Notes:
echo Download and unzip axis-bin-1_4.zip, put this wsdl2java.bat to axis-1_4\lib and run it.
echo A jar file bingads-v9.jar would be generated if all good, please use it in your app.
echo mail.jar and activiation.jar are optional, not required here.

echo -----------------------
echo Converting Ad Intelligence Service...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://api.bingads.microsoft.com/Api/Advertiser/AdIntelligence/v9/AdIntelligenceService.svc?wsdl

echo Converting Bulk Service...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://api.bingads.microsoft.com/Api/Advertiser/CampaignManagement/v9/BulkService.svc?wsdl 

echo Converting Campaign Management Service...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://api.bingads.microsoft.com/Api/Advertiser/CampaignManagement/v9/CampaignManagementService.svc?wsdl

echo Converting Customer Billing Service...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://clientcenter.api.bingads.microsoft.com/Api/Billing/v9/CustomerBillingService.svc?wsdl 

echo Converting Customer Management...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://clientcenter.api.bingads.microsoft.com/Api/CustomerManagement/v9/CustomerManagementService.svc?wsdl

echo Converting Optimizer Service...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://api.bingads.microsoft.com/Api/Advertiser/Optimizer/v9/OptimizerService.svc?wsdl 

echo Converting Reporting Service...
java -cp .;* org.apache.axis.wsdl.WSDL2Java -a https://api.bingads.microsoft.com/Api/Advertiser/Reporting/v9/ReportingService.svc?wsdl

echo Please change long[] to Long[] within the following java files to fix the ArrayOfNullableOflong issue, then run The Step 2:
echo 1. AddAdsResponse.java
echo 2. AddKeywordsResponse.java

wsdl2java-v9-step2.bat
@echo off
echo Running The Step 2......

echo Compiling java source code...
javac -cp .;* -encoding UTF-8 com/microsoft/adapi/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/bingads/AdIntelligence/v9/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/bingads/Billing/v9/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/bingads/CampaignManagement/v9/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/bingads/Customer/v9/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/bingads/Customer/v9/Entities/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/bingads/Customer/v9/Exception/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/bingads/Optimizer/v9/*.java
javac -cp .;* -encoding UTF-8 com/microsoft/bingads/Reporting/v9/*.java
javac -cp .;* -encoding UTF-8 org/datacontract/schemas/_2004/_07/Microsoft_BingAds_Advertiser_CampaignManagement_Api_DataContracts/*.java
javac -cp .;* -encoding UTF-8 org/datacontract/schemas/_2004/_07/Microsoft_BingAds_Advertiser_Optimizer_Api_DataContracts/*.java
javac -cp .;* -encoding UTF-8 org/datacontract/schemas/_2004/_07/Microsoft_BingAds_Advertiser_Optimizer_Api_DataContracts_Entities/*.java
javac -cp .;* -encoding UTF-8 org/datacontract/schemas/_2004/_07/System_Collections_Generic/*.java

echo Creating jar file bingads-v9.jar...
jar cf bingads-v9.jar com/microsoft/* org/datacontract/*

echo WSDL2Java BingAds API V9 Converting completed!

echo Use following command to add to the maven repository
echo mvn install:install-file -Dfile=bingads-v9.jar -DgroupId=com.microsoft -DartifactId=bingads -Dversion=v9 -Dpackaging=jar

Sunday, December 8, 2013

How to read the super large Excel and CSV file?

It's easy to say, I am going to use the opencsv to read CSV and POI XSSF to read Excel (XLSX). But to handle the super large data file, even tens or hundreds of gigabytes. We need to think about stream based parsers allow for ultimate performance, low resource usage, and nearly unlimited versatility scalable to any size data file, also need to bulk load and process the data file. Here is a sample code to read through data file batch by batch.

package lchen.action.datafile;

import java.util.List;

public interface IDataFileReader {
 public int rowNum(); // current row number!
 public List<String[]> readRows(int batchSize) throws Exception;
}


package lchen.action.datafile;

import java.io.FileReader;
import java.util.ArrayList;
import java.util.List;

import au.com.bytecode.opencsv.CSVReader;

public class CSVFileReader implements IDataFileReader {
 private int rowNum = 0; // current row number!
 private CSVReader reader;

 public CSVFileReader(String csvPath, char delimiter) throws Exception {
  this.reader = new CSVReader(new FileReader(csvPath), delimiter);
 }

 @Override
 public int rowNum() {
  return rowNum;
 }

 @Override
 public List<String[]> readRows(int batchSize) throws Exception {
  List<String[]> dataRows = new ArrayList<String[]>();
  if (batchSize > 0) {
   String[] dataRow = null;
   while ((dataRow = reader.readNext()) != null) {
    rowNum++;
    dataRows.add(dataRow);
    if (dataRows.size() == batchSize)
     break;
   }
  }
  return dataRows;
 }
}


package lchen.action.datafile;

import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;

import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.XMLStreamReader;

import org.apache.poi.openxml4j.opc.OPCPackage;
import org.apache.poi.openxml4j.opc.PackageAccess;
import org.apache.poi.ss.util.CellReference;
import org.apache.poi.xssf.eventusermodel.ReadOnlySharedStringsTable;
import org.apache.poi.xssf.eventusermodel.XSSFReader;
import org.apache.poi.xssf.usermodel.XSSFRichTextString;

/**
 * 
 * XSSF and XML Stream Reader
 * 
 * If memory footprint is an issue, then for XSSF, you can get at the underlying XML data, and process it yourself. This is intended for intermediate
 * developers who are willing to learn a little bit of low level structure of .xlsx files, and who are happy processing XML in java. Its relatively
 * simple to use, but requires a basic understanding of the file structure. The advantage provided is that you can read a XLSX file with a relatively
 * small memory footprint.
 * 
 * @author lchen
 * 
 */
public class XExcelFileReader implements IDataFileReader {
 private int rowNum = 0;
 private OPCPackage opcPkg;
 private ReadOnlySharedStringsTable stringsTable;
 private XMLStreamReader xmlReader;

 public XExcelFileReader(String excelPath) throws Exception {
  opcPkg = OPCPackage.open(excelPath, PackageAccess.READ);
  this.stringsTable = new ReadOnlySharedStringsTable(opcPkg);

  XSSFReader xssfReader = new XSSFReader(opcPkg);
  XMLInputFactory factory = XMLInputFactory.newInstance();
  InputStream inputStream = xssfReader.getSheetsData().next();
  xmlReader = factory.createXMLStreamReader(inputStream);

  while (xmlReader.hasNext()) {
   xmlReader.next();
   if (xmlReader.isStartElement()) {
    if (xmlReader.getLocalName().equals("sheetData"))
     break;
   }
  }
 }

 @Override
 public int rowNum() {
  return rowNum;
 }

 @Override
 public List<String[]> readRows(int batchSize) throws XMLStreamException {
  String elementName = "row";
  List<String[]> dataRows = new ArrayList<String[]>();
  if (batchSize > 0) {
   while (xmlReader.hasNext()) {
    xmlReader.next();
    if (xmlReader.isStartElement()) {
     if (xmlReader.getLocalName().equals(elementName)) {
      rowNum++;
      dataRows.add(getDataRow());
      if (dataRows.size() == batchSize)
       break;
     }
    }
   }
  }
  return dataRows;
 }

 private String[] getDataRow() throws XMLStreamException {
  List<String> rowValues = new ArrayList<String>();
  while (xmlReader.hasNext()) {
   xmlReader.next();
   if (xmlReader.isStartElement()) {
    if (xmlReader.getLocalName().equals("c")) {
     CellReference cellReference = new CellReference(xmlReader.getAttributeValue(null, "r"));
     // Fill in the possible blank cells!
     while (rowValues.size() < cellReference.getCol()) {
      rowValues.add("");
     }
     String cellType = xmlReader.getAttributeValue(null, "t");
     rowValues.add(getCellValue(cellType));
    }
   } else if (xmlReader.isEndElement() && xmlReader.getLocalName().equals("row")) {
    break;
   }
  }
  return rowValues.toArray(new String[rowValues.size()]);
 }

 private String getCellValue(String cellType) throws XMLStreamException {
  String value = ""; // by default
  while (xmlReader.hasNext()) {
   xmlReader.next();
   if (xmlReader.isStartElement()) {
    if (xmlReader.getLocalName().equals("v")) {
     if (cellType != null && cellType.equals("s")) {
      int idx = Integer.parseInt(xmlReader.getElementText());
      return new XSSFRichTextString(stringsTable.getEntryAt(idx)).toString();
     } else {
      return xmlReader.getElementText();
     }
    }
   } else if (xmlReader.isEndElement() && xmlReader.getLocalName().equals("c")) {
    break;
   }
  }
  return value;
 }

 @Override
 protected void finalize() throws Throwable {
  if (opcPkg != null)
   opcPkg.close();
  super.finalize();
 }

}
 
 

Download Source Files