laiye rpa developer guide d1-博天堂登陆

we mentioned in the chapter "a brief introduction of rpa" that one of the main features of rpa is that it is non-invasive. in other words, although rpa works with other software programs, it does not require them to provide a special interface. instead, it directly interacts with the user interface of other software programs to simulate human interactions. however, there are often many input boxes and buttons that we can interact with in a given software interface. how does laiye rpa know where and how we want to operate? the interface elements introduced in this chapter will solve this problem.

if you have some programming experiences and know what a control is, then please forget this concept for now. controls and interface elements have something in common, but they are not exactly the same, so we must try to avoid conceptual confusion.

other than programmers, most people interact with the graphical interface of the operating system when using computers. whether it is the commonly used windows and mac osx or linux, which is not well-known amongst non-programmers, they all have their own graphical interfaces. as web browsers became widely adopted, more and more graphical interfaces are displayed in a browser. these graphical interfaces can take on many shapes and forms, but when we click on them with our mouse, there is a small graphical component to register that we have clicked on something. we will call these graphical components interface elements.

for example, the figure below is a commonly seen window on the windows operating system, as well as a typical graphical user interface. what are the interface elements in this window?

figure 26: common window on windows

first of all, there are various options in the menu bar above, such as "file", "home", "share", and "view", and these are all interface elements. the icons and the text in the toolbar below, such as "copy" and "paste", are also interface elements. likewise, the "quick access", "desktop”, and "download" in the navigation bar to the left are all interface elements. each file displayed in the main area of the window (highlighted by the red box in figure 28) is also an interface element.

notice how interface elements can also contain one another. for example, the area of the window highlighted by the red box is a large interface element, and it contains the interface element representing each file inside it.

in laiye rpa, the role of the interface element is to serve as the target in commands with targets (i.e., commands that require a target).

as mentioned earlier, laiye rpa provides a large number of built-in commands, which you can find in the command area. among them, the most basic ones are as shown in figure 27.

figure 27: the most basic commands

among them, all commands under the interface element and text categories require targets. likewise, commands under the mouse and keyboard categories that contain the word "target" also require targets. these commands are highlighted by the red boxes in figure 28.

figure 28: command with target

commands with target specify an interface element as a property. when you run this command, it will first check whether this interface element exists. if it exists, the command will perform its specific operation on it. for example, if the specified interface element is a button, then the command click target will click this button. if the interface element does not exist, the command will search for the interface element repeatedly until some time limit is reached (this is known as timeout, and the specific time limit can be set as a property). an error, in turn, terminates the process.

conversely, commands without target do not take an interface element as a property. for example, the simulate click command has no target. when you run this command, it will click where the mouse currently is. as another example, the simulate key command also has no target. when running, it will simulate a key operation where the keyboard is currently focused on.

whenever possible, you should always prefer using commands with target, because they are much more accurate. only when you cannot specify the target as an interface element should you use commands without target as the second best choice.

therefore, selecting a target accurately is critical when using laiye rpa. as long as the target is selected correctly, simulating an operation is relatively simple. here is a detailed description of how to select a target.

laiye rpa provides fully automatic methods to select a target. let's consider the click target command in the mouse category as an example.

suppose we want to execute a very simple process, which only has one step: click the start menu button of windows (the default position is in the lower left corner of your screen). first, create a new process and open the only block in it. then, find the click target command in the visual view and insert it into the assembly area by dragging and dropping or double-clicking the command. you should already be familiar with these steps. if you are not, please go back and read chapter 2, "basic concepts."

now, there should be a command in the assembly area. note that there is a button on this command with the text "find target" next to a target icon, like the one in figure 29.

figure 29: click target command and the "find target" button

click this button. the laiye creator interface will be temporarily hidden, and a translucent mask with red border and blue background appears. this is the target selector, which shows you the available interface elements that you may select as the target. the target selector follows wherever the mouse moves. when we left-click on an interface element, the target selector disappears and the laiye rpa interface reappears. the interface element covered by the target selector when we left-click is the target we have selected. as mentioned earlier, interface elements may be nested, and the mouse may be hovering over multiple interface elements at the same time. in this case, the target selector will automatically choose the interface element you most likely need and cover it. therefore, before left-clicking to confirm your selection, please move your mouse patiently until the target selector perfectly covers the interface element you need.

let's have a try. use the target selector to cover the start menu button, and makes sure the selector covers it perfectly. when the mask changes to the state shown in figure 30, left-click to confirm the selection. note that the figure below is what it looks like in the windows 10. if you're using a different version of windows, what you see will be different, but the idea remains the same.

figure 30: use the "target selector" to select the start menu button

once selected, laiye creator reappears, and the find target button you just pressed now looks like a thumbnail of the target interface element you selected. this thumbnail helps you remember which target you just selected and has no impact on how the command executes. moreover, this thumbnail is actually a button. if you click on the thumbnail, it will take you to the exact same interface that you see when you click on the find target button. if the previously selected target is not what you need, or if you accidentally selected the wrong target, just press this button and try again.

notice that every command with target has a property named target. when we have not yet selected a target, the value of this property is a pair of curly brackets "{}", as shown in the left image in figure 31. note that because no target is selected, something will definitely go wrong if we try to run the command. after we have selected a target, the target property is populated with a complicated value, but it is still surrounded by a pair of curly brackets, as shown in the right image of figure 31.

figure 31: the value of the target property

the value should look something like this:

once you have learned laiye rpa's programming language, botscript, later on, you will know that the value shown here is actually a "dictionary" data type in botscript. for now, don’t worry about these details. just know that this is a special piece of data that specifies the interface elements you have selected. using this data, laiye rpa can reselect the target consistently.

if you have developed windows applications before (if you haven't, just skip to the next section), you know that there are actually many development frameworks for applications on windows, including sdk, mfc, wtl, winform, wpf, qt, java, etc. if you count web applications that run in ie and chrome, there are even more types. these frameworks all provide the capability of searching interface elements within its applications. from a technical standpoint, laiye rpa is simply referencing the interface element searching apis provided by these frameworks. this might not seem complicated, but the methods to call all these apis are different from each other, and even a skilled developer would struggle to learn all these interfaces in a short time, let alone users not familiar with software development.

however, if you use laiye rpa, all of these different frameworks are all abstracted as the same interface elements, and you can search and operate on them in the same way. for example, a button in an mfc program and a button in chrome may seem like they are all just buttons, but the technical difference in simulating clicking on these two buttons is huge. in laiye rpa, you don't need to care about these differences at all, because laiye rpa will handle the details for us. therefore, this is how laiye rpa achieves and balances the three indicators of an rpa platform: powerful, simple, and fast.

we have learned in the previous section that the target selector in laiye rpa works automatically. as long as we move the mouse to the interface element that we want as the target, the target selector will perfectly mask over the needed interface element and generate a piece of data to describe the element, which allows laiye rpa to find the target again during runtime.

naturally, any automatic tools will fail from time to time. when using target selectors, there are some common problems:

• no matter how we move the mouse, we can't make the mask perfectly cover the intended interface element. a common situation is that the mask is covering the whole window, instead of selecting the element you need, which might be smaller or inside that window.

• the mask can perfectly cover the interface elements, but when you try to find the target during runtime using the generated data, one of the following occurs:

- incorrect selection: the interface element can be found, but it is not the one we originally selected.

- missed selection: the interface element existed at the time of selection, but now it cannot be found.

we will address the first scenario, i.e., when the target cannot be covered, in detail in the next chapter. here, we will focus on the second scenario, i.e., when the target can be perfectly covered, but laiye rpa selects incorrectly or fails to select the target in runtime. as we mentioned in the previous section, laiye rpa generates a piece of data similar to the one below to describe a selected target.

laiye rpa finds the target according to this piece of data when running the process. therefore, when laiye rpa misses or incorrectly selects a target, something is wrong with this data, and we need to modify it.

locate the target property of the command, where the piece of data is displayed in the corresponding input box. it is possible to directly edit the content of the field using the input box, but the box is too small for efficient editing. therefore, we recommend you click on the button to the right of the input box, shown in red in figure 32.

figure 32: modify target

after you click the button, a target editor window pops up. the image on the top of the window is a thumbnail, which shows the general appearance of the target interface element. once again, this is just for your own reference, and laiye rpa will not use it when running the process. right below the thumb is the control filter, which displays the piece of data you saw earlier as a tree structure (figure 33).

figure 33: target editor

this tree structure displays the features of the interface element chosen by the target selector earlier. each item is a feature, and they have been automatically selected by laiye rpa to describe the element. during runtime, laiye rpa only selects an interface element if it contains all these features. moreover, because the interface elements are nested with each other, laiye rpa will not only record the features of the target interface element, but also the features of parent interface elements of the target. therefore, for an element to be selected, the features of each specified level must be satisfied.

take the start menu button of windows in the above figure as an example. the line 0: object and the entries underneath it represent the features of the interface element that contains the start menu button (the windows taskbar), while the line 1: object and the entries underneath it represent the features of the start menu button itself. during runtime, laiye rpa will search level by level. it will first find the windows taskbar at the first level; then it will search for the start menu button within the children elements of the taskbar.

such strict feature matching naturally leads to situations when laiye rpa is unable to find an element that fits. in this example, windows assigns start menu a "title," with the value "start" (the title is usually hidden from users, but it exists underneath the hood). laiye rpa takes this title as part of its feature, because, in general, the title of the button does not change. however, if one day the title of this button changes, laiye rpa would fail to find the button.

in this case, we’ll need to modify our feature matching process. we can see that there is a check box before every feature that is checked by default. uncheck the "title" field, and laiye rpa will no longer use this feature when searching for the target interface element. this way, even if the title changes, the target interface element can still be found.

however, if too many features are removed, though you are less likely to miss the intended target, you are more likely to end up with an incorrect selection. for example, if you uncheck both cls: "start" and title: "start", then any interface element under the windows taskbar (selected by 0: object) can meet the condition, and laiye rpa will likely select the wrong element.

therefore, if the interface elements are complicated, or if their features often change, then it takes some practice and skill to edit the target to accurately select the intended element. unfortunately, there are no specific rules in this regard, so you need to learn from experience. here are several general guidelines to help you get started:

• you may be unfamiliar with some feature names for the time being, such as cls, aaname, etc. don’t worry about them.

• make good use of the wildcard symbol *, which matches anything. for example, if there is an interface element that has a feature title with value "name: james". the actual name “james” might change, but the first part of the title “name: “ will remain constant. therefore, you can use title: "name: *" as a feature, rather than simply removing it.

• be conservative when removing features. although removing features can reduce missed selections, your feature set could quickly become too general to avoid selecting the wrong elements. when the process is running, missed selections are generally easier to debug, while incorrect selections may not be found immediately.

expanding on the last point, we note that when laiye rpa is running a process, most commands with target (except for commands like decide if target exists) throw an exception when the described target element is not found. unless the exception is handled by a try…catch clause (more on this later), it would terminate the process immediately and report the error to you, which makes it easy to notice. on the other hand, if laiye rpa selects an incorrect target, it does not know that, so it will continue to run, which silently suppresses the error and makes it hard to notice.

lastly, the same interface elements may have different features on different operating systems and different browsers. especially for internet explorer and chrome, the same elements on the same page usually have drastically different features. for example, selecting the search bar on  in ie and chrome generates massively different feature sets (see figure 34).

figure 34: the features of the same target are different on ie and chrome browsers

therefore, when using the laiye rpa to create a process to be run on other computers, please try to ensure that the development and production environments are consistent (i.e., same operating system, same browser, etc.) in order to reduce unnecessary runtime errors.

the list of built-in commands interacting with interface elements is given in figure 35.

figure 35: the list of interface element operations

don’t be overwhelmed by how long this list is. there is no need to learn all these commands one by one. by learning some of the commands and understanding the design of laiye rpa, you can extrapolate problem solving principles that help you know what commands and what even properties for these commands to use when you have a certain functionality in mind.

as we have learned, operations on interface elements are executed through commands, and properties are an indispensable part of the command. in fact, a command is complete only when its key properties are clearly specified. generally speaking, a command consists of the following parts: command = (do what) (to what) (with what) to (get what).

do what refers to operation itself. in general, the types of operations that can be performed on a target depends on the type of the target. once you have determined the target, you know what operations you can do to it. for example, you can use click on a button, while you generally can’t click on texts. in a similar vein, there are special operations to checkboxes, including one determining whether a box is checked and one setting the status of a checkbox. there are also some common operations that are available in almost all interface elements, such as obtaining the size, position, and text of an interface element.

to what refers to the object, or target, of the command’s operation. for interface elements, targets can be dropdowns, checkboxes, textboxes, and more. we have already learned how to select an interface element using laiye rpa’s target selector.

with what refers to additional modifiers the user needs to provide when invoking this command. this also depends on the type of the target. for example, the user needs to enter a boolean value (true or false) when setting the status of a checkbox. however, there are some commands that can be executed without any additional information from the user, such as get element text. as long as the user sets the target of this command, the command can get the text of the element without any more information. in this respect, "with what" is similar to parameters in programming. a command that does not require additional information is just like a function that takes no parameter.

get what refers to the result returned by the command. for example, invoking the command get element text returns a string whose value is the text of the element. likewise, invoking the command get element selection against a set of checkboxes returns an array of boolean values (each value corresponds to one checkbox).

in essence, laiye rpa simulates real human operation to computers. it encapsulates each operation into a command, and the key properties of these commands are determined by its target.

let us consider a test form to demonstrate how to operate interface elements. this interface includes text labels, radio boxes, checkboxes, single-line textboxes, multi-line text boxes, list boxes, dropdowns, etc., which are all common interface elements in forms.

figure 36: test form

as the name implies, the command check if element exists determines if a specific interface element exists in the current screen and stores the result in a variable. when the interface element exists, the command returns true; otherwise it returns false.

if you need to check whether a desired interface has appeared on the screen, you can check if a key element on that interface exists using this command. if this element exists, it indicates that the interface appears; otherwise it means that the interface does not exist.

for example, let's check whether the submit button on the test form exists.

1. drag and drop the check if element exists command to the visual view, as shown in figure 37.

   
   
   figure 37: check if element exists

2. click the find target button on check if element exists, switch to the test form, and hover your mouse over the submit button, so that the red edge and blue mask perfectly covers the submit button. left-click to confirm your selection.
   

3. the result of the check if element exists command is saved in the variable bret, as shown in figure 37. we can use an conditional to check whether the result is true or false. true indicates that the element is found, and false indicates that the element is not found.
   

the command set/get element ticked allows you to operate form elements, specifically radio boxes and checkboxes, automatically. let's use the test form again for demonstration.

1. drag and drop the set element ticked command to the assembly area. use the search target button to select the options "music" and "sport" under the "hobby" section. take care to select the corresponding checkboxes, not the texts "music" and "sport" themselves.

2. drag and drop the get element ticked command to the assembly area. use the search target button to select the options "music" and "sport" under the "hobby" section.

3. use the output command to display the values of bret and bret2, which correspond to whether the selected checkboxes are checked.

4. run the process (see figure 38). you should see "true" twice, which indicates that both elements are checked.
   

   
   figure 38 : the command and result of setting/getting element ticked
   

5. on the testing page, we can see that the checkboxes for "music" and "sport" are now checked. we have successfully filled the form using these two commands.

when selecting a target with a mask, we often run into situations where the mask only ever covers an entire window, and no matter what you do, the mask refuses to select the interface elements within the window. as shown in figure 39, the mask only selects the whole chrome tab, but it cannot select the input boxes and buttons inside the tab.

figure 39: trouble selecting specific elements in a chrome tab

there are two possible reasons for this situation. one possibility is that there is no possible way to select this interface element (more on this later). the other possibility is that the interface is actually selectable, but it requires additional support from laiye rpa. at the time of writing, laiye rpa provides extensions for chrome, firefox, and java programs. installing the corresponding extensions allows us to select page elements in chrome, page elements in firebox, and java interface elements like swing, awt, jnpl, applet, etc.

let's take the chrome browser as an example to see how we can install these extensions.

chrome requires us to both install and enable the laiye rpa extension to properly select elements inside a webpage. after you have installed the extension, you will see the icon in figure 40, located to the right of the address bar. when you hover over the icon, a tooltip with the text "laiye rpa native message plugin" should appear.

figure 40: laiye rpa extension icon on chrome browser

if you have not installed the extension, follow these steps to install it:

1. close chrome.

2. open any process in laiye rpa creator. select "help" -> "install extension" in the menu, and then select "chrome extension".

3. open chrome. the browser should prompt you that a new extension has been added (figure 41). click "enable extension”. you should be all set.
   

   
   figure 41: chrome browser prompts you that a new extension has been added
   

4. if you are still experiencing difficulties selecting chrome interface elements, please follow figure 42 to enable laiye rpa native message plugin.
   

   
   figure 42: extension management of chrome browser