RestTemplate with Google Places API
jramoyo
7:31 PM
cache
,
checkthecrowd
,
google places
,
guava
,
rest
,
resttemplate
,
spring
,
web service
1 comment
My website, CheckTheCrowd.com, was initially using the Google Maps JavaScript API (Places Library) to fetch details on the various places submitted to the website.Place details such as names, addresses, and photos are normally displayed as content. Because these contents were dynamically loaded via JavaScript, they won't be visible to web crawlers and hence cannot be read as keywords.
In order for web crawlers to access the place details, they needed to be included as part of the HTML generated from by the Servlet. This meant that rather than fetch the place details from the browser via JavaScript, I needed to fetch them from the web server.
Place Details - Google Places API
Under the hood, the JavaScript Places Library calls a REST service to fetch the details of a particular place. I needed to call the same service from the web server in order to deliver the place details as part of the Servlet content.
The Place Details REST service is a GET call to the following resource:
https://maps.googleapis.com/maps/api/place/details/output?parametersWhere output can either be JSON (json) or XML (xml), the resource requires 3 parameters: the API key (key), the place identifier (reference), and the sensor flag (sensor).
For example:
https://maps.googleapis.com/maps/api/place/details/json?reference=12345&sensor=false&key=54321
RestTemplate - Spring Web
Starting with version 3.0, the Spring Web module comes with a class called RestTemplate. Similar to other Spring templates, RestTemplate reduces boiler-plate code that is normally involved with calling REST services.
RestTemplate supports common HTTP methods such as GET, POST, DELETE, PUT, etc. Objects passed to and returned from these methods are converted by HttpMessageConverters. Default converters are registered against the MIME type and custom converters are also supported.
RestTemplate and Place Details
RestTemplate exposes a method called getForObject to support GET method calls. It accepts a String representing the URL template, a Class for the return type, and a variable String array to populate the template.
I started my implementation by creating a class called GooglePlaces. I then declared the URL template as a constant and declared RestTemplate as an instance member injected by the Spring container. My Google Places API key was also declared as a member instance, this time populated by Spring from a properties file:
private static final String PLACE_DETAILS_URL = "https://maps.googleapis.com/" + "maps/api/place/details/json?reference=" + "{searchId}&sensor=false&key={key}"; @Value("${api.key}") private String apiKey; @Inject private RestTemplate restTemplate;The above code should be enough to call the Place Details service and get the response as JSON string:
String json = restTemplate.getForObject(PLACE_DETAILS_URL, String.class, "12345", apiKey);However, the JSON response needs to be converted to a Java object to be of practical use.
By default, RestTemplate supports JSON to Java conversion via MappingJacksonHttpMessageConverter. All I need is to do is create Java objects which map to the Place Details JSON response.
Java Mapping
I referred to Place Details reference guide for a sample of the JSON response that I needed to map to Java. Because the Place Details response includes other information that I didn't need for CheckTheCrowd, I added annotations to my classes which tells the converter to ignore unmapped properties:
@JsonIgnoreProperties(ignoreUnknown = true) public static class PlaceDetailsResponse { @JsonProperty("result") private PlaceDetails result; public PlaceDetails getResult() { return result; } public void setResult(PlaceDetails result) { this.result = result; } }The above class represents the top-level response object. It is simply a container for the result property.
The below class represents the result:
@JsonIgnoreProperties(ignoreUnknown = true) public static class PlaceDetails { @JsonProperty("name") private String name; @JsonProperty("icon") private String icon; @JsonProperty("url") private String url; @JsonProperty("formatted_address") private String address; @JsonProperty("geometry") private PlaceGeometry geometry; @JsonProperty("photos") private List<PlacePhoto> photos = Collections.emptyList(); // Getters and setters... }I also needed the longitude and latitude information as well as the photos. Below are the classes for the geometry and photo properties which contain these information:
@JsonIgnoreProperties(ignoreUnknown = true) public static class PlaceGeometry { @JsonProperty("location") private PlaceLocation location; public PlaceLocation getLocation() { return location; } public void setLocation(PlaceLocation location) { this.location = location; } }
@JsonIgnoreProperties(ignoreUnknown = true) public static class PlaceLocation { @JsonProperty("lat") private String lat; @JsonProperty("lng") private String lng; // Getters and setters... }
@JsonIgnoreProperties(ignoreUnknown = true) public static class PlacePhoto { @JsonProperty("photo_reference") private String reference; public String getReference() { return reference; } public void setReference(String reference) { this.reference = reference; } }With the above Java mappings, I can now expose a method which returns an instance of PlaceDetails given a place reference:
public PlaceDetails getPlaceDetails(String searchId) { PlaceDetailsResponse response = restTemplate.getForObject(PLACE_DETAILS_URL, PlaceDetailsResponse.class, searchId, apiKey); if (response.getResult() != null) { return response.getResult(); } else { return null; } }
Caching
The moment I deployed my changes to Tomcat, I noticed a significant latency between server requests. This was expected because the server now has to make several calls to the Place Details service before returning a response.
This is exactly a scenario where a good caching strategy would help. It is worth noting however that Google Maps API Terms of Service (10.1.3.b) has strict rules regarding caching. It states that caching should only be done to improve performance and that data can only be cached up to 30 calendar days.
CheckTheCrowd uses Guava which includes a pretty good API for in-memory caching. Using a CacheLoader, I can seamlessly integrate a Guava cache to my code:
private LoadingCache<String, PlaceDetails> placeDetails = CacheBuilder.newBuilder() .maximumSize(1000) .expireAfterAccess(24, TimeUnit.HOURS) .build(new CacheLoader<String, PlaceDetails>() { @Override public PlaceDetails load(String searchId) throws Exception { PlaceDetailsResponse response = restTemplate.getForObject(PLACE_DETAILS_URL, PlaceDetailsResponse.class, searchId, apiKey); if (response.getResult() != null) { return response.getResult(); } else { throw new PlacesException("Unable to find details for reference: " + searchId); } } });I set a cache size of 1000 and an expiry of 24 hours. The call to the Place Details service was then moved to the CacheLoader's load method. After which, I updated my method to refer to the cache instead:
public PlaceDetails getPlaceDetails(String searchId) { try { return placeDetails.get(searchId); } catch (ExecutionException e) { logger.warn("An exception occurred while " + "fetching place details!", e.getCause()); return null; } }
The complete source is available from Google Code under Apache License 2.0.
Generating Sitemaps Using Spring Batch and SitemapGen4j
jramoyo
11:53 AM
batch job
,
checkthecrowd
,
chunk processing
,
dynamic
,
java
,
jsp
,
sitemap
,
spring
,
spring batch
6 comments
I recently launched a website called CheckTheCrowd.com. And in order for search engines to effectively crawl my content, I needed a sitemap.Since my content is mostly generated from the database, I needed to find a way to dynamically generate my sitemap.
Most answers I got from online forums suggested exposing a URL which when accessed, generates the sitemap. With Spring MVC, it goes something like:
@RequestMapping("/sitemap.xml") public @ResponseBody String generateSitemap() { String sitemap = generateExpensiveXml(); return sitemap }The problem with this approach is that it doesn't scale. The more content you have, the longer it takes to generate the sitemap. And because the sitemap is generated every time the URL is accessed, precious server resources are wasted.
Another suggestion was to append an entry to the sitemap every time new content is added to the database. I did not like this approach because it would be difficult to do source control on the sitemap. Also, accidentally deleting the sitemap would mean that data is gone forever.
Batch Job Approach
Eventually, I ended-up doing something similar to the first suggestion. However, instead of generating the sitemap every time the URL is accessed, I ended-up generating the sitemap from a batch job.
With this approach, I get to schedule how often the sitemap is generated. And because generation happens outside of an HTTP request, I can afford a longer time for it to complete.
Having previous experience with the framework, Spring Batch was my obvious choice. It provides a framework for building batch jobs in Java. Spring Batch works with the idea of "chunk processing" wherein huge sets of data are divided and processed as chunks.
I then searched for a Java library for writing sitemaps and came-up with SitemapGen4j. It provides an easy to use API and is released under Apache License 2.0.
Requirements
My requirements are simple: I have a couple of static web pages which can be hard-coded to the sitemap. I also have pages for each place submitted to the web site; each place is stored as a single row in the database and is identified by a unique ID. There are also pages for each registered user; similar to the places, each user is stored as a single row and is identified by a unique ID.
A job in Spring Batch is composed of 1 or more "steps". A step encapsulates the processing needed to be executed against a set of data.
I identified 4 steps for my job:
- Add static pages to the sitemap
- Add place pages to the sitemap
- Add profile pages to the sitemap
- Write the sitemap XML to a file
Step 1
Because it does not involve processing a set of data, my first step can be implemented directly as a simple Tasklet:
public class StaticPagesInitializerTasklet implements Tasklet { private static final Logger logger = LoggerFactory.getLogger(StaticPagesInitializerTasklet.class); private final String rootUrl; @Inject private WebSitemapGenerator sitemapGenerator; public StaticPagesInitializerTasklet(String rootUrl) { this.rootUrl = rootUrl; } @Override public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception { logger.info("Adding URL for static pages..."); sitemapGenerator.addUrl(rootUrl); sitemapGenerator.addUrl(rootUrl + "/terms"); sitemapGenerator.addUrl(rootUrl + "/privacy"); sitemapGenerator.addUrl(rootUrl + "/attribution"); logger.info("Done."); return RepeatStatus.FINISHED; } public void setSitemapGenerator(WebSitemapGenerator sitemapGenerator) { this.sitemapGenerator = sitemapGenerator; } }The starting point of a Tasklet is the execute() method. Here, I add the URLs of the known static pages of CheckTheCrowd.com.
Step 2
The second step requires places data to be read from the database then subsequently written to the sitemap.
This is a common requirement, and Spring Batch provides built-in Interfaces to help perform these types of processing:
- ItemReader - Reads a chunk of data from a source; each data is considered an item. In my case, an item represents a place.
- ItemProcessor - Transforms the data before writing. This is optional and is not used in this example.
- ItemWriter - Writes a chunk of data to a destination. In my case, I add each place to the sitemap.
For this step, I declare a JdbcCursorItemReader in my Spring configuration and set my implementation of RowMapper:
@Bean public JdbcCursorItemReader<PlaceItem> placeItemReader() { JdbcCursorItemReader<PlaceItem> itemReader = new JdbcCursorItemReader<>(); itemReader.setSql(environment.getRequiredProperty(PROP_NAME_SQL_PLACES)); itemReader.setDataSource(dataSource); itemReader.setRowMapper(new PlaceItemRowMapper()); return itemReader; }Line 4 sets the SQL statement to query the ResultSet. In my case, the SQL statement is fetched from a properties file.
Line 5 sets the JDBC DataSource.
Line 6 sets my implementation of RowMapper.
Next, I write my implementation of ItemWriter:
public class PlaceItemWriter implements ItemWriter<PlaceItem> { private static final Logger logger = LoggerFactory.getLogger(PlaceItemWriter.class); private final String rootUrl; @Inject private WebSitemapGenerator sitemapGenerator; public PlaceItemWriter(String rootUrl) { this.rootUrl = rootUrl; } @Override public void write(List<? extends PlaceItem> items) throws Exception { String url; for (PlaceItem place : items) { url = rootUrl + "/place/" + place.getApiId() + "?searchId=" + place.getSearchId(); logger.info("Adding URL: " + url); sitemapGenerator.addUrl(url); } } public void setSitemapGenerator(WebSitemapGenerator sitemapGenerator) { this.sitemapGenerator = sitemapGenerator; } }Places in CheckTheCrowd.com are accessible from URLs having this pattern: checkthecrowd.com/place/{placeId}?searchId={searchId}. My ItemWriter simply iterates through the chunk of PlaceItems, builds the URL, then adds the URL to the sitemap.
Step 3
The third step is exactly the same as the previous, but this time processing is done on user profiles.
Below is my ItemReader declaration:
@Bean public JdbcCursorItemReader<PlaceItem> profileItemReader() { JdbcCursorItemReader<PlaceItem> itemReader = new JdbcCursorItemReader<>(); itemReader.setSql(environment.getRequiredProperty(PROP_NAME_SQL_PROFILES)); itemReader.setDataSource(dataSource); itemReader.setRowMapper(new ProfileItemRowMapper()); return itemReader; }Below is my ItemWriter implementation:
public class ProfileItemWriter implements ItemWriter<ProfileItem> { private static final Logger logger = LoggerFactory.getLogger(ProfileItemWriter.class); private final String rootUrl; @Inject private WebSitemapGenerator sitemapGenerator; public ProfileItemWriter(String rootUrl) { this.rootUrl = rootUrl; } @Override public void write(List<? extends ProfileItem> items) throws Exception { String url; for (ProfileItem profile : items) { url = rootUrl + "/profile/" + profile.getUsername(); logger.info("Adding URL: " + url); sitemapGenerator.addUrl(url); } } public void setSitemapGenerator(WebSitemapGenerator sitemapGenerator) { this.sitemapGenerator = sitemapGenerator; } }Profiles in CheckTheCrowd.com are accessed from URLs having this pattern: checkthecrowd.com/profile/{username}.
Step 4
The last step is fairly straightforward and is also implemented as a simple Tasklet:
public class XmlWriterTasklet implements Tasklet { private static final Logger logger = LoggerFactory.getLogger(XmlWriterTasklet.class); @Inject private WebSitemapGenerator sitemapGenerator; @Override public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception { logger.info("Writing sitemap.xml..."); sitemapGenerator.write(); logger.info("Done."); return RepeatStatus.FINISHED; } }Notice that I am using the same instance of WebSitemapGenerator across all the steps. It is declared in my Spring configuration as:
@Bean public WebSitemapGenerator sitemapGenerator() throws Exception { String rootUrl = environment.getRequiredProperty(PROP_NAME_ROOT_URL); String deployDirectory = environment.getRequiredProperty(PROP_NAME_DEPLOY_PATH); return WebSitemapGenerator.builder(rootUrl, new File(deployDirectory)) .allowMultipleSitemaps(true).maxUrls(1000).build(); }Because they change between environments (dev vs prod), rootUrl and deployDirectory are both configured from a properties file.
Wiring them all together...
<beans> <context:component-scan base-package="com.checkthecrowd.batch.sitemapgen.config" /> <bean class="...config.SitemapGenConfig" /> <bean class="...config.java.process.ConfigurationPostProcessor" /> <batch:job id="generateSitemap" job-repository="jobRepository"> <batch:step id="insertStaticPages" next="insertPlacePages"> <batch:tasklet ref="staticPagesInitializerTasklet" /> </batch:step> <batch:step id="insertPlacePages" parent="abstractParentStep" next="insertProfilePages"> <batch:tasklet> <batch:chunk reader="placeItemReader" writer="placeItemWriter" /> </batch:tasklet> </batch:step> <batch:step id="insertProfilePages" parent="abstractParentStep" next="writeXml"> <batch:tasklet> <batch:chunk reader="profileItemReader" writer="profileItemWriter" /> </batch:tasklet> </batch:step> <batch:step id="writeXml"> <batch:tasklet ref="xmlWriterTasklet" /> </batch:step> </batch:job> <batch:step id="abstractParentStep" abstract="true"> <batch:tasklet> <batch:chunk commit-interval="100" /> </batch:tasklet> </batch:step> </beans>Lines 26-30 declare an abstract step which serves as the common parent for steps 2 and 3. It sets a property called commit-interval which defines how many items comprises a chunk. In this case, a chunk is comprised of 100 items.
There is a lot more to Spring Batch, kindly refer to the official reference guide.
Testing Spring MVC Annotated JSON Controllers from JUnit
Update: While I still recommend this post for good reading, please refer to this post for a better approach at doing this.My previous post explained how we can use AnnotationMethodHandlerAdapter to test annotations applied to Spring MVC Controllers. This post will attempt to explain how we can reuse the classes introduced in the previous post to test Spring Controllers that return a JSON response.
The @ResponseBody annotation allows Spring Controllers to define the contents of an HTTP response.
For this example, we assume that Spring MVC is configured to represent the contents of the HTTP response as JSON.
Let's say we have a controller which returns user information from a GET request:
@Controller public class MyJsonController { private final UserService userService = new UserService(); @RequestMapping(value = "/user/{username}", method = RequestMethod.GET) public @ResponseBody User getUser(@PathVariable String username) { return userService.getUser(username); } // Mocked for illustration purposes private static class UserService { public User getUser(String username) { User user = new User(username); user.setFirstName("Jan"); user.setLastName("Amoyo"); return user; } } }This controller returns a User object serialized within the body of the HTTP response (in our case, as JSON). Because it doesn't return an instance of ModelAndView, we cannot use the previously introduced AbstractControllerTest to test the output.
In order to test the JSON response, we need to assign an appropriate HttpMessageConverter to the AnnotationMethodHandlerAdapter defined in AbstractControllerTest. The one we need is MappingJacksonHttpMessageConverter.
Extending AbstractControllerTest
We will create a new class called AbstractJsonControllerTest and extend from AbstractControllerTest. Here, we override the parent's constructor so that we can assign MappingJacksonHttpMessageConverter to the AnnotationMethodHandlerAdapter. We also add various convenience methods to help process JSON.
@Ignore("abstract test case") public abstract class AbstractJsonControllerTest<T> extends AbstractControllerTest<T> { private final ObjectMapper mapper; public AbstractJsonControllerTest(T controller) { super(controller); mapper = new ObjectMapper(); MappingJacksonHttpMessageConverter jacksonHttpMessageConverter = new MappingJacksonHttpMessageConverter(); MappingJacksonHttpMessageConverter[] messageConverters = { jacksonHttpMessageConverter }; getHandlerAdapter().setMessageConverters(messageConverters); } protected List<Map<String, Object>> convertJsonArrayHttpServletResponseToList(MockHttpServletResponse response) throws JsonParseException, JsonMappingException, IOException { return convertJsonArrayStringToList(response.getContentAsString()); } protected List<Map<String, Object>> convertJsonArrayStringToList(String json) throws JsonParseException, JsonMappingException, IOException { return mapper.readValue(json, new TypeReference<List<HashMap<String, Object>>>() { }); } protected Map<String, Object> convertJsonHttpServletResponseToMap(MockHttpServletResponse response) throws JsonParseException, JsonMappingException, IOException { return convertJsonStringToMap(response.getContentAsString()); } protected Map<String, Object> convertJsonStringToMap(String json) throws JsonParseException, JsonMappingException, IOException { return mapper.readValue(json, new TypeReference<HashMap<String, Object>>() { }); } protected String convertObjectToJsonString(Object object) throws JsonMappingException, JsonGenerationException, IOException { return mapper.writeValueAsString(object); } }Similar to AbstractControllerTest, we can use AbstractJsonControllerTest as the parent class to all test cases involving JSON Spring Controllers.
Example
Here is how we test MyJsonController:
@RunWith(BlockJUnit4ClassRunner.class) public class MyJsonControllerTest extends AbstractJsonControllerTest<MyJsonController> { public MyJsonControllerTest() { super(new MyJsonController()); } @Test public void testGetUser() throws Exception { getRequest().setMethod("GET"); getRequest().setServletPath("/user/jramoyo"); ModelAndView modelAndView = invokeController(); assertNull(modelAndView); String jsonString = getResponse().getContentAsString(); assertFalse(jsonString == null || jsonString.isEmpty()); Map<String, Object> jsonMap = convertJsonStringToMap(jsonString); assertEquals("jramoyo", jsonMap.get("username")); assertEquals("Jan", jsonMap.get("firstName")); assertEquals("Amoyo", jsonMap.get("lastName")); } }Line 13 asserts that ModelAndView is null as expected.
Line 16 asserts that the JSON reponse is not null nor empty.
Line 18 converts the JSON response to a Map object.
Lines 19 to 21 asserts the correctness of the user information retrieved from the HTTP response.
The above snippets are available from Google Code.
Testing Spring MVC Annotated Controllers from JUnit
Let me begin by clarifying that this post is more about testing the correctness of annotations applied to Spring Controllers rather than unit testing the behavior of the controller methods.Annotations on Spring Controllers are commonly tested manually, together with the views. Aside from the usual concerns that come with it, manual testing also involves significant overhead with
having to redeploy binaries and restarting the web server in-between fixes. While "hot-deploy" reduces a lot of these concerns, it still leaves some room for improvement.
AnnotationMethodHandlerAdapter
Starting with version 2.5, Spring MVC comes with a class called AnnotationMethodHandlerAdapter which can be used to programatically invoke an annotated Spring Controller. The handle() method from this class accepts an HttpServletRequest, an HttpServletResponse, and an Object representing the Controller to invoke. It returns an instance of ModelAndView.
We can use AnnotationMethodHandlerAdapter within a JUnit test case to simulate a web container handling a request to the Spring Controller.
@Test public void testController() throws Exception { ModelAndView modelAndView = handlerAdapter.handle(request, response, myController); assertEquals("index", modelAndView.getViewName()); }We can mock HttpServletRequest and HttpServletResponse using popular mocking frameworks such Mockito or use the built-in MockHttpServletRequest and MockHttpServletResponse from the Spring Test module. This example uses the built-in mocked objects from Spring Test.
Using AnnotationMethodHandlerAdapter together with Generics, we can create an abstract class which serves as a parent class to all Spring Controller unit test cases:
@Ignore("abstract test case") public abstract class AbstractControllerTest<T> { private final T controller; private final AnnotationMethodHandlerAdapter handlerAdapter; private MockHttpServletRequest request; private MockHttpServletResponse response; public AbstractControllerTest(T controller) { handlerAdapter = new AnnotationMethodHandlerAdapter(); this.controller = controller; } @Before public void setUp() { response = new MockHttpServletResponse(); request = new MockHttpServletRequest(); } protected T getController() { return controller; } protected AnnotationMethodHandlerAdapter getHandlerAdapter() { return handlerAdapter; } protected MockHttpServletRequest getRequest() { return request; } protected MockHttpServletResponse getResponse() { return response; } protected ModelAndView invokeController() throws Exception { return handlerAdapter.handle(request, response, controller); } }As a parent class, it takes care of the boiler plate code involved in setting-up and invoking the controller.
Example
Let's say we have a controller which returns a view named "index" and populates the current user's username as a model attribute:
@Controller public class MyController { @RequestMapping(value = "/", method = RequestMethod.GET) public String getHomePage(Principal principal, Model model) { if (principal != null) { model.addAttribute("username", principal.getName()); } return "index"; } }To test this, we create a concrete subclass of AbstractControllerTest and assign MyController as the generic type:
@RunWith(BlockJUnit4ClassRunner.class) public class MyControllerTest extends AbstractControllerTest<MyController> { public MyControllerTest() { super(new MyController()); } @Test public void testGetHomePage() throws Exception { getRequest().setMethod("GET"); getRequest().setServletPath("/"); getRequest().setUserPrincipal(new HttpPrincipal("username", "realm")); ModelAndView modelAndView = invokeController(); assertEquals("index", modelAndView.getViewName()); assertNotNull(modelAndView.getModelMap().get("username")); assertEquals("username", modelAndView.getModelMap().get("username")); } }Line 6 creates an instance of MyController and passes it as a parameter to the parent class' constructor.
Lines 11 to 13 sets-up the HTTP request by assigning the request method, request URL, and a fake Principal.
Line 15 invokes the request against the Spring Controller.
Lines 16-18 asserts the correctness of the returned ModelAndView.
The above snippets are available from Google Code.
On my next post, I will explain how to test Spring Controllers returning a JSON response by extending the above classes.
Subscribe to:
Posts
(
Atom
)
1 comment :
Post a Comment