Accepted Manuscript Title: Development of a tactile sensor based on optical fiber specklegram analysis and sensor data fusion technique Authors: Eric Fujiwara, Yu Tzu Wu, Murilo Ferreira Marques dos Santos, Egont Alexandre Schenkel, Carlos Kenichi Suzuki PII: DOI: Reference:
S0924-4247(17)30379-5 http://dx.doi.org/doi:10.1016/j.sna.2017.07.031 SNA 10226
To appear in:
Sensors and Actuators A
Received date: Revised date: Accepted date:
5-3-2017 8-7-2017 16-7-2017
Please cite this article as: Eric Fujiwara, Yu Tzu Wu, Murilo Ferreira Marques dos Santos, Egont Alexandre Schenkel, Carlos Kenichi Suzuki, Development of a tactile sensor based on optical fiber specklegram analysis and sensor data fusion technique, Sensors and Actuators: A Physicalhttp://dx.doi.org/10.1016/j.sna.2017.07.031 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Development of a tactile sensor based on optical fiber specklegram analysis and sensor data fusion technique Eric Fujiwaraa*, Yu Tzu Wua, Murilo Ferreira Marques dos Santosa, Egont Alexandre Schenkela, Carlos Kenichi Suzukia a
Laboratory of Photonic Materials and Devices, School of Mechanical Engineering,
University of Campinas, Campinas, 13083-860 Brazil.
e-mails: [email protected]
, [email protected]
*Corresponding author ([email protected]
A tactile sensor based on fiber specklegram analysis and data fusion was developed.
Measurements are performed over 30 30 mm² area by utilizing only 3 fibers.
The sensor has 0.5 N-1 sensitivity and detects changes of 1 mm in force location.
The force magnitude and spatial distribution can be estimated by data fusion.
Abstract The development of a tactile sensor based on optical fiber specklegram analysis is reported. The device is comprised of 9 microbending transducers connected to 3 multimode fibers, and attached to a 30 30 mm² touching surface in a matrix arrange. The output fiber speckle fields, produced by coherent light transmission through the multimode waveguides, are processed for evaluation of the normalized inner-product coefficients, being further correlated to the external load characteristics according to a specklegram referencing approach. Finally, the magnitude and location of the forces applied over the tactile frame are estimated by means of a data fusion technique, yielding the probabilistic distribution of
the mechanical disturbances. The sensor characterization indicated a 0.5 N-1 sensitivity, with capability to detect variations of 1 mm in force location, being comparable to the requirements for tactile technologies. Moreover, the data fusion approach provided the estimation of the load distribution with 0.86 relative accuracy, allowing the assessment of tactile variables based on the response of only 3 sensing fibers and without utilizing wavelength multiplexing schemes.
Keywords: tactile sensor, optical fiber sensor, fiber specklegram sensor, sensor data fusion, force measurement. 1. Introduction Tactile sensors are defined as devices capable of obtaining information about an object or environment by means of physical contact . Such devices can be utilized for perceiving several types of physical variables, such as applied forces, vibration, and temperature, as well as object features including texture, shape, and geometry ,. In association with vision capability, the ability of tactile perception in humans plays an important role on the accomplishment of manipulation and exploration tasks, allowing the subject to handle tools with different levels of force and dexterity, and to perceive the external media based on physical feedback. In this sense, the development of technologies capable of detecting and processing tactile variables is essential for the implementation of advanced systems, such as dexterous manipulators and humanoid robots , medical instruments for minimally invasive surgery , healthcare and rehabilitation systems , agriculture and food processing , and user interfaces . On the other hand, the replication of tactile ability involves several technological challenges, as the contact devices must be capable of retrieving the temporal and spatial information distributed over the sensor area, demanding the processing of large datasets ,. Additionally, it is necessary to distribute massive sensing elements along a sensing area with minimal spacing, which implies a tradeoff between spatial resolution and measurement range . Furthermore, as the device is usually subjected to direct mechanical excitation, the sensor structure must be durable, resistant to the environment, and preferably flexible, passive, and present a low implementation cost .
Currently, a variety of tactile sensors have been developed based on different transducing principles. For example, piezoresistive devices are typically designed as arrayed strain gauges made of silicon chip diaphragms  or conductive polymers  and provide good linearity and high mechanical flexibility, but their characteristics are strongly affected by temperature . Capacitive sensors are manufactured by interposing a pair of conductive grid layers with a dielectric film , presenting high sensitivity to normal forces, despite the complexity of interrogation circuits . Tactile matrices for medical and robotics applications based on the piezoelectric effect of polymers have also been reported , but their usage is restricted to dynamic excitations . Another approach consists in using inductive devices based on electrical or magnetic interrogation , allowing one to retrieve the forces directions and realize frequency multiplexing, although such implementation demand complex schemes . Finally, reflection-type optical sensors can be fabricated from arrays of LED-photodetector pairs  or flexible planar waveguides , making it possible to obtain the forces distribution in a direct form. However, such devices can present limitations in terms of hysteresis and spatial resolution . In this context, optical fiber sensors figure as a promising alternative for designing practical tactile interfaces. These devices can be applied for monitoring a wide range of physical variables, and present inherent characteristics such as high sensitivity, lightweight, distributed measurements and multiplexing capabilities, and immunity to electromagnetic interference . Extrinsic-type sensors can be obtained by using a fiber bundle to collect the light reflected in a flexible surface , or deformable elements with embedded anglepolished end face waveguides for mechanical modulation of coupled light intensity . Even though the operation principle is straightforward, the optical response depends on the precise alignment of each fiber, which can be difficult to implement in case of dense matrixes. Another example consists in forming a grid of sensing points by interweaving multiple waveguides , resulting in light attenuation by microbending losses. Such approach can also be applied for designing smart textiles with embedded plastic optical fibers , which are convenient for producing wearable tactile systems. In spite of the linearity and sensitivity provided by microbending transducers, the monitoring of several points require the simultaneous measurement of an excessive number of fibers, increasing the hardware costs and processing time. Alternatively, the assessment of distributed forces
can also be carried out by fiber Bragg grating (FBG) systems, by arranging the sensing elements in a tactile array – or embedding the fibers in flexible foils to create artificial skins . The main feature of this approach is the ability to perform wavelengthdivision multiplexing, allowing one to retrieve the information from multiple gratings with a few sensing fibers. On the other hand, FBG sensors usually require expensive interrogation schemes, and can not resolve successive points with spatial resolution of a few millimeters. In this sense, present paper reports the development of a tactile matrix based on optical fiber specklegram sensors (FSS). The FSS are based on the analysis of the output speckle fields (specklegrams) , providing an evaluation of the fiber status deviations in response to an external stimulus, and working as a single-arm interferometer. Such sensors exhibit high sensitivity, relative low implementation cost, and multiplexing capability , being applied on the measurement of displacements , temperature , chemical concentration , and biomedical signals . Therefore, the proposed tactile sensor utilizes the fiber status information encoded in the speckle fields to establish a grid of force transducers, with further processing of acquired optical signals by means of a probabilistic grid-like data fusion method.
2. Measurement principle 2.1. Fiber specklegram sensor Optical fibers specklegram sensors are based on the analysis of the speckle fields projected from the fiber end face, produced due to the propagation of coherent light through a multimode waveguide . When the fiber is subjected to mechanical or thermal disturbances, the configuration of the output intensity peaks is varied in response to changes on the light guidance conditions . Since the speckle field retains information about the fiber status, it is possible to measure the external stimulus by quantifying the specklegram deviations, which can be accomplished in a very sensitive way by means of correlation functions. Given the intensity I(x,y) of a speckle field projected over a xy plan,  defined the normalized inner-product coefficient (NIPC) of specklegrams, denoted by NIPC I ( x , y) I 0 ( x , y) dx dy
I (x, y) dx dy I (x, y) dx dy 2
where I0(x,y) is the intensity for a reference fiber status. According to Eq. (1), the NIPC value decreases from 1 to ~0 as I deviates from I0. In case of force F measurements, one may address I0 to the undisturbed condition (F = 0), and the magnitude of applied force can be retrieved from NIPC value after prior calibration.
2.2. Specklegram referencing The specklegram analysis can also provide information about the external stimuli applied on different locations of the same fiber, which is a key feature for tactile sensors. For example, consider the arrange formed by a pair of force transducers (namely TA and TB) spaced by a certain distance and attached to the same waveguide, as illustrated in Fig. 1(a). Each device is subjected to an independent force (FA or FB) ranging from 0 to a maximum value (Fmax), causing the modulation of a single output specklegram. In terms of fiber statuses, the speckle field intensity I(x,y) can assume 4 characteristic patterns according to the combination of applied forces, defining 4 statuses as shown in Fig. 1(b). Initially, the fiber is under an undisturbed neutral condition N, corresponding to the absence of external stimuli (FA = 0, FB = 0). If FA is increased to the maximum value, the specklegram will be changed to another configuration due to light attenuation and redistribution of the speckle pattern, reaching the status A (FA = Fmax, FB = 0). Since the specklegram is reversible under controlled conditions, removing the force FA will cause I(x,y) to be restored to the status N. Moreover, if FA is gradually increased, the speckle pattern will undergo partial modifications proportional to the force magnitude, but it will always converge to A when FA = Fmax. The same behavior can be observed by exciting transducer TB, leading the specklegram to another fiber status B (FA = 0, FB = Fmax). Finally, if the waveguide is disturbed by both transducers simultaneously with maximum load, the output pattern will assume a fourth configuration AB (FA = Fmax, FB = Fmax), which can also be reached from A or B by increasing FB or FA, respectively. Therefore, the fiber status can be fully described in terms of applied forces (Fmax F 0) by these 4 conditions (N, A, B, and AB).
Fig. 1. Simultaneous force sensing scheme based on specklegrams correlation analysis: (a) measurement setup with 2 transducers (TA and TB), excited by forces FA and FB, respectively, and attached on a single multimode fiber (MMF). The light emitted by a laser source (LS) is guided through the MMF, yielding an output speckle field intensity I(x,y); (b) diagram of possible fiber statuses (N, A, B, AB) due to force modulation.
In order to quantify the specklegram deviations, it is convenient to calculate 4 NIPC values by addressing the reference condition I0 to each defined status. As abovementioned, the inner-product coefficient indicates the degree of deviation between different speckle fields, consequently the NIPC assumes unitary value only if the current status matches the reference one. Even though the NIPC decreases as the fiber is modulated by the external stimulus, if the specklegram changes to another predefined status, the inner-product for this second reference will increase proportionally to the applied force, making it possible to estimate the magnitude and relative location of the measured variable by combining the information obtained from each NIPC curve.
3. Materials and methods 3.1. Tactile sensor design The schematic of the tactile sensor is shown in Fig. 2. The device comprises a flexible 30 30 mm², 0.05 mm thickness polyvinyl chloride (PVC) plate, which works as the touch surface, mounted over a rigid acrylic substrate. Microbending transducers formed by graphite rods (10 mm length and 0.5 mm diameter) arranged in periodical fashion (5 mm periodicity) are placed in the internal faces of the plates, constituting the corrugated deformer structures. The bending device encloses a section of the ~2 m length silica
multimode optical fibers (62.5 m diameter cladding with polymer buffer and planepolished end face), causing the mechanical modulation of transmitted light intensity . In order to provide spatial sensitivity to the tactile frame, the microbending transducers are symmetrically distributed over the sensor area in a 3 3 matrix arrange, being each row crossed by one of the 3 optical fibers. This setup was obtained based on previous results , presenting an improved generalization capability. The relative position is defined by a Cartesian coordinate system (u,v) centered in the sensor surface, Fig. 2(a), being the transducers denoted by T(u0,v0), where (u0,v0) corresponds to the centre of the microbending device.
Fig. 2. Optical fiber tactile sensor: (a) top and (b) side views. MMF: multimode fiber (k denotes the kth fiber); (u,v): coordinate system of the sensor frame; T (u0,v0): microbending transducers; F(u,v): applied force. A photograph of the tactile sensor is shown in (c).
3.2. Experimental setup The experimental setup is presented in Fig. 3. The light emitted by a continuous HeNe laser source (633 nm) is coupled into the multimode fibers by means of a launching stage. The waveguides are firstly subjected to a mode scrambler for modal power distribution balancing, being further connected to the tactile matrix as shown in Fig. 2. The output light is detected by an uEye IDS UI-2230SE-C-HQ CCD camera with 1/3" size
color sensor, 1024 768 pixels resolution, and 15 fps frame rate, equipped with a f/1.4 aperture, 8 mm focal distance lens. Then, the specklegram images are acquired and processed through routines developed under MATLAB (Mathworks) environment.
Fig. 3. Experimental setup. LS: laser source; MS: mode scrambler; MMF: multimode fiber; F(u,v): applied force; FS: output specklegrams; CCD: detector; PC: data processing and analysis.
3.3. Data processing The specklegrams images in color space are firstly converted to grayscale and then filtered by 2D wavelet transform (Daubechies db4) for noise removal. Next, each captured frame proceeds to the NIPC calculation, Eq. (1), by addressing the reference intensity I0(x,y) to the different fiber statuses obtained by prior calibration. For the tactile array comprised of 9 microbending transducers T(u0,v0) and 3 optical fibers (Fig. 2), it is convenient to define the reference conditions in order to evaluate the overall force distribution over the sensor area. Given a punctual force F(u,v), the NIPC computed for the kth fiber and referenced to F(u0,v0) = Fmax status is denoted by Nk(u,v), whereas the NIPC for the undisturbed condition F(u0,v0) = 0 is identified by Nk0. In this sense, one may choose F(u0,v0) for the locations occupied by each one of the 9 microbending transducers, resulting in 3 sets of 10 NIPC curves. Consequently, the force spatial distribution can be estimated by combining the data obtained from each waveguide.
3.4. Sensor data fusion
The specklegram referencing approach can be extended for the case of arrayed transducers, as observed in tactile sensors, by choosing additional reference statuses for evaluating the force response at different locations. However, as the grid size increases, it becomes necessary to process a considerable number of NIPC curves in order to correctly decode the spatial information from the speckle patterns. Moreover, establishing the calibration curves for every possible fiber status may be laborious and difficult to implement in practical setups. In this sense, a feasible alternative consists in obtaining a limited set of NIPC curves referenced to single load configurations, and then estimating the overall force distribution based on the combination of available data. Although this procedure can be implemented by means of artificial neural networks (ANN) , it is difficult to generalize the ANN output in the space domain, yielding a force response inscribed in a fixed-size grid. The information acquired from multiple transducers can be processed by utilizing a data fusion technique based on Bayesian inference , in which the NIPC curves are treated as individual measurement elements that are combined to form a multisensor network. According to the Bayes' theorem, the conditional probability function P(x|z) described by the states vector x for the observations z is given by
P (x | z) P (z | x) P (x) P (z) ,
where P(z|x) is the observations model, P(x) is the prior probability, and P(z) is the marginal probability . Regarding the data fusion problem, the posterior probability P(x|z) can be obtained from the sensor model P(z|x), whereas the prior knowledge P(x) of the states vector is firstly assumed, and then adjusted by recursive update. Finally, P(z) = is set as a normalizing constant. The sensor model provides the force distribution P(u,v) computed from the NIPC values. For a single transducer T(u0,v0) assessed by means of the kth fiber specklegram, the related force distribution Pk(u0,v0)(u,v) can be described in terms of a Gaussian function,
Pk (u0 ,v 0 ) (u, v)
0 ,v 0 )
u u0 2 v v0 2 exp , 2 k2(u0 ,v 0 ) 2
where (u0,v0) is the transducer location, k(u0,v0) is the standard deviation, and is an offset. Since the probability function profile indicates the force probability around (u0,v0), k(u0,v0) must be modulated according to the magnitude of applied load on transducer T(u0,v0), which
is obtained from the corresponding NIPC value Nk(u0,v0). Consequently, once the correlation between k(u0,v0) and Nk(u0,v0) is determined by static calibration, the force distribution Pk(u0,v0)(u,v) can be calculated as a function of the NIPC deviation. On the other hand, it is also convenient to analyze the information provided by Nk0 for identifying if the tactile sensor is on absence of applied forces. In such cases, the Gaussian curve can be modulated to a flat profile for simulating an uniform distribution, indicating that the kth fiber is not subjected to direct load. Therefore, one may define the standard deviation as
0 ,v 0 )
f (Nk (u0 ,v0 ) ) g(Nk 0 ) ,
where f and g are modulation functions, mainly defined by the calibration curves and tactile frame dimensions. According to the multisensor data fusion approach , the force distribution probability P(u,v) is calculated by
P (u, v) P0 (u, v) Pk(u0 ,v0 ) (u, v) . k
Eq. (5) combines the information contained in every sensor model Pk(u0,v0) in order to generate the posterior probability P(u,v) based on the prior knowledge P0(u,v), which can be assumed as an uniform distribution or set to the current load condition in case of recursive updating . The constant must be adjusted for each iteration in order to normalize the force distribution P(u,v). Finally, since the Gaussian function decays to zero as the position (u,v) deviates from the center (u0,v0), it is important to set the offset to a non-zero value for preventing Pk(u0,v0) to assume a null probability for any location, which could cancel the contribution of a particular element in Eq. (5).
4. Sensor characterization 4.1. Force response For assessing the static sensitivity of the fiber sensor, a mechanical stage equipped with load cell was utilized to input a controlled force on the tactile frame center F(0,0). Initially, the actuator 2.5 mm spherical-shaped probe is carefully adjusted to maintain a minimum contact with the sensor surface with negligible force. Next, the magnitude of
mechanical excitation is increased to 1.5 N in steps of 0.1 N, by keeping the load constant for 5 s per increment. The NIPC was evaluated for each fiber with reference to the undisturbed condition (N10, N20, and N30), as indicated in Fig. 4, where each point represents the average of 15 experiments, considering the mean of a 5 s acquisition interval per data value. The NIPC values tend to decrease with the increase of applied force as expected, since the fiber status deviates from the reference one. N20 presented a linear behavior for the 0 F 0.6 N range, followed by a saturation for F 1 N. On the other hand, N10 and N30 exhibited a low sensitivity region for F 0.5 N, with a further improvement in the sensor response as the modulation force increases. The absolute static sensitivities within the linear ranges for N10, N20, and N30 are 0.22 N-1, 0.50 N-1, and 0.33 N-1, respectively. The best result was obtained for N20 because the transducer T(0,0) (attached to optical fiber 2) is located immediately under the mechanical excitation source, whereas the adjacent transducers are modulated by the PVC plate deformation. Since the displacements caused by the frame bending are relatively small for reduced forces, the transducers attached to fibers 1 and 3, T(0,10) and T(0,10), respectively, produce only a slight modulation of the light guidance condition for F 0.5 N, causing the low sensitivity range in N10 and N30, which is typical of microbending sensors . However, in case of the present setup, such behavior is convenient for discriminating the response of each NIPC curve and retrieving the location of applied force, as the effect of the punctual excitation in the tactile frame center is preferable to be observed only for fiber 2. The N20 value is also affected by the contribution of adjacent transducers T(10,0) and T(10,0), causing the fiber 2 to be excited by multiple microbending sources even with the application of a punctual force. The 0.5 N-1 sensitivity (~0.02 N resolution) for N20 is also compatible to guideline values proposed for tactile sensors . In case of N10 and N30, even though both curves present the same trend, the NIPC for fiber 1 present a lower sensitivity than the later one, probably due to variations in the sensor construction. It is worth noticing that the specklegram sensors sensitivity is strongly affect by the fiber core dimension and the refractive indexes difference. The number of modes NM supported by a MMF is proportional to the core diameter, numerical aperture, and the freespace wave number . Since NM is related to the number of visible light speckles, the
utilization of waveguides with larger cores provides the enhancement of the sensor sensitivity, as most of specklegram processing techniques are based on the quantification of subtle changes between the output speckle images obtained for different fiber statuses. Consequently, the increase on the number of supported modes produces a more granular intensity distribution in the detector plane, yielding an improved pixel-by-pixel differentiation in response to the input stimulus variations . Therefore, a possible approach for sensitivity enhancement would consist in utilizing polymer optical fibers, since plastic waveguides can be fabricated with larger core dimensions, and also present higher mechanical strength and flexibility , being suitable to force measurements. Regarding the dynamic range, the saturation of N20 is due to the high divergence between I(x,y) and I0(x,y), induced by modal phasing deviations . Despite the limited range compromises the utilization of this technology in certain applications, the 0–0.6 N interval is comparable to piezoresistive and capacitive sensors . Moreover, the measurement range can be improve by resetting the reference status I0 in order to shift the calibration curve for higher force values , or by specklegram partitioning .
Fig. 4. Sensor response to a punctual force F(0,0) applied on the center of tactile frame. The average NIPC values were adjusted by cubic polynomial fitting (R² > 0.99).
4.2. Spatial response The sensor spatial response was investigated by applying a constant 0.5 N force and moving the actuator probe over the tactile frame surface by means of a micrometric stage.
The experiments were carried out by aligning the probe with (0,0) position and scanning the sensor surface along u and v directions separately, considering a 10 to 10 mm range with steps of 1 mm. Finally, the NIPC values were calculated by assuming Fmax = 0.5 N and referencing Nk(u0,v0) to the transducers comprehended in the probe path. The results for u and v-axes are shown in Fig. 5(a) and Fig. 5(b), respectively, with each curve representing the average of 10 experiments with 5 s acquisition interval per point. In case of Fig. 5(a), the NIPC were calculated only for k = 2, as the other fibers are not crossed by the force path (v = 0), even though the modulation of the adjacent waveguides may be not negligible given that the upper plate deformation induces stress on the neighboring transducers. For F(u,0) constant, as the probe is moved away from a microbending device, the respective NIPC value tend to decrease in response to the speckle pattern changes, resulting in the increase of adjacent N2(u,0) since I approaches to its reference status. Although N2(10,0), N2(0,0), and N2(10,0) are correlated because the corresponding transducers modulate the same fiber, it is still possible to identify the relative force location based on the interrogation of a single speckle field. Moreover, intermediary positions (not considered during the calibration) can be assessed by combining the data from different N2(u,v) curves. Concerning the v-axis scanning, Fig. 5(b), the analyses were conducted for N1(0,10), N2(0, 0), and N3(0,10) as the 3 fibers are crossed by the probe path. The results are analogous to the previous case, demonstrating that the methodology allows force location estimation in transverse direction. Eventual differences in the characteristics of the NIPC curves are probably due to variations in the physical construction of the microbending transducers, affecting the form in which the specklegrams change in response to the external stimulus. Nevertheless, it is possible to detect variations of ~1 mm in the force location by combining the dynamic ranges of different NIPC curves, which is compatible to the 1–2 mm spatial resolution suggested by tactile sensing guidelines .
Fig. 5. Sensor response to the application of a 0.5 N force on different locations of the tactile frame over (a) u and (b) v-axes. The experimental data were adjusted by 4th degree polynomial fitting (R² > 0.95).
5. Tactile measurements For evaluating the tactile sensing capability of proposed methodology, the system is initially subjected to a calibration procedure, with application of constant force (~1 N) onto each transducer location for ~5 s. Next, the recorded specklegrams are utilized as references for calculating the Nk(u0,v0) and Nk0 curves, being further processed by the data fusion algorithm in order to retrieve the force probability distribution P(u,v) according to Eq. (5). The determination of the modulation functions f and g was carried out based on the static calibration curves (Fig. 4) with empirical adjusting. Assuming that the relationship between F and Nk(u0,v0) is linear within the 0.5 Nk(u0,v0) 1 interval for the sake of simplicity, and choosing Nk0 = 0.7 as a threshold value for a full load condition, the standard deviation can be defined as a linear function of Nk(u0,v0),
k (u ,v ) 16 12 N k (u ,v 0
Equation (6) ensures that the force distribution Pk(u0,v0) assumes narrower shape when the current fiber status matches the reference condition Nk(u0,v0), and the output range of k(u0,v0) values was determined by simulation to maximize the spatial generalization capability of the methodology. Finally, the prior probability P0 was specified as an uniform distribution, whereas the offset was set to 0.01 to keep any Pk(u0,v0) from vanishing.
In the first experiment, the sensor response was tested for the application of a single ~1 N load on equally spaced positions over the tactile frame. The touch surface was scanned along the u and v-axes, covering most of the sensor area, by keeping the load for ~5 s in each position and repeating the experiments 5 times per measurement setup. The calculated force distributions for different load conditions are shown in Fig. 6, with the nominal force locations (u*,v*) indicated by asterisks. It is observed that the P(u,v) value increases in the vicinities of the load source, indicating a higher probability of finding a mechanical disturbance on the related area. As expected, the P(u,v) curves presented sharper profiles for (u*,v*) = (u0,v0), corresponding to the cases predicted on the calibration set. Moreover, the force location was estimated with considerable probability for intermediary (u*,v*) positions, even though the P(u,v) profiles exhibited a broadening tendency as a consequence of the data fusion procedure, since the load distribution was estimated from the response of adjacent transducers. In order to quantify the relative error of sensor response, the force probability at the nominal locations P(u*,v*) was calculated, as P(u,v) would assume unitary value when exactly matches the load position, (u,v) = (u*,v*), providing an indirect measurement of the sensor accuracy. The results are shown in Table 1, regarding the average of 5 experiments. It is worth noting that the probability values were normalized during the data fusion step. The trend of P(u*,v*) values corroborates with Fig. 6, indicating a higher accuracy for the locations comprised in the calibration dataset. The average values are 0.930.05 and 0.810.14 for the calibration and intermediary positions, respectively, yielding an overall accuracy of 0.860.12, which can be considered a reliable result given the utilization of only 3 sensing fibers. An alternative for improving the sensor response consists in tuning up the modulation functions f and g in order to reduce the spreading of P(u,v) profile and match its maximum value to the nominal location. Considering that the microbending transducers can present differences on their physical construction, one may define individual modulation functions based on the characteristics of each device for better equalization of the Pk(u0,v0)(u,v) distributions. In addition, the standard deviation k(u0,v0) can be also modulated as a function of the direction, which would be useful for compensating the interaction between adjacent transducers, especially for attenuating the undesirable broadening of P(u,v) in case of neighboring fibers, as observed in Fig. 6. Finally, the sensor
accuracy can be refined with the adjustment of prior probability P0(u,v) in Eq. (5) by recursive update, allowing the probability around the P(u,v) peak for each iteration to increase, which is useful regarding practical measurements with time-correlated data acquisition. However, it is important to notice that the force probability will not necessarily converge to the nominal location if P(u,v) is misestimated due to the incorrect conditioning of the modulation functions, as the central tendency will be not shifted for new iterations.
Fig. 6. Force probability distribution P(u,v) evaluated over the sensor frame for single load conditions. The colors indicate the P(u,v) value, whereas the asterisks show the nominal location of applied force.
Table 1. Force probability values at nominal load locations P(u*,v*). The higher the P(u*,v*) value, the more accurate is the force position estimation. v* (mm) 10 5 0 -5 -10
-10 0.910.01 0.790.06 0.920.01 0.750.08 0.850.05
-5 0.880.04 0.900.06 0.770.10
u* (mm) 0 0.970.01 0.690.13 1.000.00 0.960.05 0.990.00
5 0.930.02 0.980.03 0.730.13
10 0.940.02 0.720.11 0.880.03 0.610.09 0.930.02
Next, for investigating the effect of force magnitude on P(u,v), the tactile matrix was subjected to arbitrary loads ranging from 0 to ~1 N by keeping the static force for ~5 s, with verification of nominal values by means of a load cell. The experiments were carried out by choosing u* = 10, 5, and 0 mm and maintaining v* = 0 mm, with 5 repetitions per measurement condition. Concerning the effect of F(u*,0) on the P(u,0) curve profile, Fig. 7(a), Fig. 8(a), and Fig. 9(a), the probability distribution peak converges to the nominal load location with the increase of force magnitude, whereas low F(u*,0) values cause P(u,0) to assume spread distribution around (0,0). It is important to observe that the baseline of P(u,0) curves was subtracted for the sake of visualization, as the normalization constant brings the load probability distribution peak to the unitary value. The behavior of P(u,0) is due to the characteristics of modulation functions on k(u0,v0) in Eq. (6), since the force magnitude induces changes on the specklegram configuration, thus yielding variations in Nk(u0,v0) and Nk0 values. For F(u*,0) 0.4 N, the increase of k(u0,v0) causes the overlapping of the individual force distributions Pk(u0,v0)(u,v) and produces a higher probability at the tactile frame center during the data fusion procedure. In this case, the accuracy on force localization could be improved by changing the modulation functions f and g in the sense of enhancing the NIPC sensitivity within the low F(u*,0) range. Regarding the intermediary position, Fig. 8(a), one may notice that the characteristic of P(u,0) curve tended to a bimodal distribution for F(5,0) = 0.8 N instead of producing a well-centered peak at the force location. It occurred because the adjacent probabilities Pk(10,0)(u,v) and Pk(0,0)(u,v) are modulated to sharp profiles for higher Nk(u0,v0) values, yielding the separation of each peak after the data fusion.
The relationship between P(u*,v*) and F(u*,v*) is presented in Fig. 7(a), Fig. 8(a), and Fig. 9(a). It is possible to observe that P(u*,v*) increases proportionally to the magnitude of applied force, presenting saturation regions for F(u*,v*) 0.2 N and F(u*,v*) 0.7 N, probably caused by the convergence of the P(u,v) distributions. In particular, the peaks separation for F(5,0) also generated a turning point at F(u*,v*) 0.7 N. The sensitivities within the linear range are 1.43, 1.83, and 1.89 N-1 for u* = 10, 5, and 0 mm, respectively.
Fig. 7. Effect of applied force magnitude on the sensor response for (u*,v*) = (10,0) mm: (a) baselineremoved force probability distributions P(u,0); and (b) correlation between P(u*,v*) and force magnitude, with experimental data adjusted by cubic polynomial fitting.
Fig. 8. Effect of applied force magnitude on the sensor response for (u*,v*) = (5,0) mm: (a) baselineremoved force probability distributions P(u,0); and (b) correlation between P(u*,v*) and force magnitude, with experimental data adjusted by cubic polynomial fitting.
Fig. 9. Effect of applied force magnitude on the sensor response for (u*,v*) = (0,0) mm: (a) baseline-removed force probability distributions P(u,0); and (b) correlation between P(u*,v*) and force magnitude, with experimental data adjusted by cubic polynomial fitting.
The analysis of P(u,v) and P(u*,v*) curves suggests the possibility of estimating the magnitude and location of applied forces. Firstly, the position can be inferred in a probabilistic way by calculating the modes of P(u,v) distribution, returning the possible spatial range for the load occurrence. Next, the magnitude is obtained after removing the baseline of P(u,v) and then evaluating P(u*,v*) for the probable (u*,v*) values, allowing F(u,v) to be retrieved from previous calibration. However, due to the probabilistic nature of the results, it may be difficult to obtain the exact location of the load source. Even though such characteristic can compromise the spatial resolution of the sensor, in practical tactile measurements the load is usually applied in distributed form, as the dimensions of the probe (the human finger tip, for example) are comparable to the tactile frame size, making the probabilistic approach suitable for such applications. Evidently, the spatial response also can be improved by reducing the distance between the microbending elements for better interpolation of intermediary locations. On the other hand, there are some implications regarding the number of transducers installed on a single fiber, since the speckle field will be severely attenuated due to the accumulated bending losses, impeding the computation of the NIPC values. In spite of the uncertainties related to the spatial response, the proposed setup allows one to identify the applied force characteristics with acceptable response. Regarding the application of simultaneous loads, Fig. 10 illustrates the force probabilities for the tactile matrix subjected to a pair of ~1 N forces at dissimilar locations.
For the sensor excited along the v-axis, P(u,v) tends to assume a bimodal characteristic, resulting in higher probabilities on the vicinities of the input loads. A similar behavior can be observed when the forces are applied in diagonally opposed locations, allowing the discrimination of each mechanical stimuli. Conversely, the evaluation of u direction yielded a high probability distribution over the 10 u 10 mm range, making it difficult to estimate the correct locations of the forces. Such effect is due to the fact that the involved transducers are attached to the same optical fiber (k = 2), as the simultaneous excitation cause the specklegram to be shifted to another fiber status. Since such combined condition was not predicted on the calibration set, both N2(u0,v0) and N20 tend to decrease, causing the the probability function to spread. As N10 and N30 are still recognized as partial loading conditions because fibers 1 and 3 are not directly affected by the applied forces, the data fusion procedure is weighted for the intermediary fiber, producing the P(u,v) profile observed in Fig. 10. Consequently, the assessment of multiple loads may require additional implementations, such as the installation of fibers in transverse direction, or the evaluation of NIPC referenced to complementary statuses.
Fig. 10. Force probability distribution P(u,v) evaluated over the sensor frame for multiple load conditions. The colors indicate the P(u,v) value, whereas the asterisks show the nominal location of applied force.
Even though the results were repeatable for the presented measurement conditions, the sensor overall response can change due to environmental effects, such as vibration, humidity, and temperature variations, since the speckle field formation is sensible to
microscopic modulations of the fiber refractive index . Moreover, experimental parameters such as the laser stability, launching setup, detector resolution, and the fiber end face condition can also affect the light speckles visibility , inducing deviations on the sensor sensitivity and dynamic range. Fortunately, once the trends of the calibration curves are not expected to change as they depend only on the applied forces, the tactile sensor can be simply recalibrated by updating the reference fiber statuses and adjusting the fusion algorithm through Eq. (6). Concerning practical setups, it is necessary to mount the sensor structure on a stable support, and to confine the guiding fiber sections inside a rigid, vibration-free enclosure in order to avoid extraneous light modulation and restrict the physical interactions to the tactile surface.
6. Conclusion A tactile sensor based on optical fiber specklegram analysis was successfully developed. The static characterization experiments revealed a 0.5 N-1 sensitivity, with possibility to detect variations of 1 mm in the force location. Moreover, the combination of speckle field processing with data fusion approach allowed the estimation of the magnitude and spatial distribution of the mechanical stimuli over the 30 30 mm² sensor area, even with the utilization of a reduced number of optical fibers. The proposed device can be implemented with a relative simple interrogation scheme, with possibility of further utilization as a pressure or contact sensor in human-system interaction applications. In comparison to the commercial devices, the presented sensor can be fabricated using inexpensive materials and standard multimode fibers for telecommunications, whereas the interrogation system can be simplified with the utilization of a semiconductor laser and a webcam. Moreover, the data fusion approach can be extended to different types of fiber sensors (including FBG and intensity-based setups) in order to enhance their generalization capability and minimize the number of required waveguides. On the other hand, additional developments are still necessary to improve the spatial resolution and to properly handle the distributed load scenarios, which can be accomplished
by including complementary fibers, addressing additional reference statuses, or using polymer fibers with larger core dimension and numerical aperture for improving the sensitivity and the speckle fields differentiation. Moreover, the sensor must be mounted in a stable structure in order to isolate the guiding fibers from extraneous vibration, and periodical calibrations are necessary to compensate eventual temperature and laser stability variations. Therefore, the authors are currently working on the optimization of the tactile sensor characteristics, as well as on the design of a real-time measurement setup for practical applications.
Funding: This work was supported by Sao Paulo Research Foundation (FAPESP) [grant number 2014/25080-0], CNPq, and CAPES.
M.H. Lee, H.R. Nicholls, Tactile sensing for mechatronics—a state of the art survey, Mechatronics 9 (1999) 1–31.
M.I. Tiwana, S.J. Redmond, N.H. Lovell, A review of tactile sensing technologies with applications in biomedical engineering, Sensor. Actuat. A-Phys. 179 (2012) 17–31.
H. Yousef, M. Boukallel, K. Althoefer, Tactile sensing for dexterous in-hand manipulation in robotics—a review, Sensor. Actuat. A-Phys. 167 (2011) 171–187.
M.E.H. Eltaib, J.R. Hewit, Tactile sensing technology for minimal access surgery—a review, Mechatronics 13 (2003) 1163–1177.
S.C. Mukhopadhyay, Wearable sensors for human activity monitoring: a review, IEEE Sens. J. 15 (2015) 1321–1330.
H-K. Kim, S. Lee, K-S. Yun, Capacitive tactile sensor array for touch screen application, Sensor. Actuat. A-Phys. 165 (2011) 2–7.
R.S. Dahiya, G. Metta, M. Valle, G. Sandini, Tactile sensing—from humans to humanoids, IEEE T. Robot. 26 (2009) 1–20.
L.D. Harmon, Automated tactile sensing, Int. J. Robot. Res. 1 (1982) 3–32.
H. Takao, M. Yawata, K. Sawada, M. Ishida, A multifunctional integrated silicon tactile imager with arrays of strain and temperature sensors on single crystal silicon diaphragm, Sensor. Actuat. A-Phys. 160 (2010) 69–77.
 G. Canavese, S. Stassi, C. Fallauto, S. Corbellini, V. Cauda, V. Camarchia, M. Pirola, C.F. Pirri, Piezoresistive flexible composite for robotic tactile applications, Sensor. Actuat. A-Phys. 208 (2014) 1–9.  F. Castelli, An integrated tactile-thermal robot sensor with capacitive tactile array, IEEE Trans. Ind. Appl. 38 (2002) 85–90.  R.S. Dahiya, D. Cattin, A. Adami, C. Collini, L. Barboni, M. Valle, L. Lorenzelli, R. Oboe, G. Metta, F. Brunetti, Towards tactile sensing system on chip for robotic applications, IEEE Sens. J. 11 (2011) 3216–3226.  H. Nakamoto, M. Goka, S. Takenawa, Y. Kida, Development of tactile sensor using magnetic elements, in: Proc. IEEE Work. Robotic Intelligence in Informationally Structured Space, 2011, pp. 37–42.  G. De Maria, C. Natale, S. Pirozzi, Force/tactile sensor for robotic applications, Sensor. Actuat. A-Phys. 175 (2012) 60–72.  J-H. Lee, C-H. Won, High-resolution tactile imaging sensor using total internal reflection and nonrigid pattern matching algorithm, IEEE Sens. J. 11 (2011) 2084–2093.  B. Culshaw, Optical fiber sensor technologies: opportunities and—perhaps— pitfalls, J. Lightwave Technol. 22 (2004) 39–50.  J.L. Schneitter, T.B. Sheridan, An optical tactile sensor for manipulators, Robot. Com-Int. Manuf. 1 (1984) 65–71.
 A. Ataollahi, P. Polygerinos, P. Puangmali, L.D. Seneviratne, K. Althoefer, Tactile sensor array using prismatic-tip optical fibers for dexterous robotic hands, in: Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, 2010, pp. 910–915.  S.R. Emge, C-L. Chen, Two-dimensional contour imaging with a fiber optic microbend tactile sensor array, Sensor. Actuat. B-Chem. 3 (1991) 31–42.  J-S. Heo, C-H. Han, J-J. Lee, System design and evaluation of the robot tactile sensor using the microbending fiber optic sensors, in: Proc. IEEE Int. Conf. Robot & Human Interactive Communication, 2007, pp. 14–18.  M. Rothmaier, M.P. Luong, F. Clemens, Textile pressure sensor made of flexible plastic optical fibers, Sensors 8 (2008) 4318–4329.  J-S. Heo, J-H. Chung, J-J. Lee, Tactile sensor arrays using fiber Bragg grating sensors, Sensor. Actuat. A-Phys. 126 (2006) 312–327.  B.M. Cowie, D.J. Webb, B. Tam, P. Slack, P.N. Brett, Fiber Bragg grating sensors for distributive tactile sensing, Meas. Sci. Technol. 18 (2007) 138–146.  L.H. Negri, E.M. Schiefer, A.S. Paterno, M. Muller, J.L. Fabris, An approach to improve the spatial resolution of force mapping sensing system, Meas. Sci. Technol. 27 (2016) 025103 (8pp).  J. Missine, B.V. Hoe, Artificial skin based on flexible optical tactile sensors, SPIE Newsroom. (2010) 1–3.  F.T.S. Yu, M. Wen, S. Yin, C-M. Uang, Submicrometer displacement sensing using inner-product multimode fiber speckle fields, Appl. Optics 32 (1993) 4685– 4689.  L. Rodriguez-Cobo, M. Lomer, J-M. Lopez-Higuera, Fiber specklegrammultiplexed sensor, J. Lightwave Technol. 33 (2015) 2591–2597.  T. Okamoto, I. Yamaguchi, Multimode fiber-optic Mach-Zehnder interferometer and its use in temperature measurement, Appl. Optics 27 (1988) 3085–3087.  E. Fujiwara, Y.T. Wu, C.K. Suzuki, Vibration-based specklegram fiber sensor for measurement of properties of liquids, Opt. Laser Eng. 50 (2012) 1726–1730.
 E. Fujiwara, Y.T. Wu, M.F.M. Santos, E.A. Schenkel, C.K. Suzuki, Optical fiber specklegram sensor for measurement of force myography signals, IEEE Sens. J. 17 (2017) 951–958.  N. Takai, T. Asakura, Statistical properties of laser speckles produced under illumination from a multimode optical fiber, J. Opt. Soc. Am. 2 (1985) 1282–1290.  N. Lagakos, J.H. Cole, J.A. Bucaro, Microbend fiber-optic sensor, Appl. Optics 26 (1987) 2171–2180.  E. Fujiwara, F.D. Paula, Y.T. Wu, M.F.M. Santos, E.A. Schenkel, C.K. Suzuki, Optical fiber tactile sensor for user interfaces, in: Proc. IEEE Int. Symp. Consumer Electronics, 2016, pp. 11–12.  E. Fujiwara, F.D. Paula, Y.T. Wu, M.F.M. Santos, E.A. Schenkel, C.K. Suzuki, Optical fiber tactile sensor based on fiber specklegram analysis, in: Proc. SPIE 10323, 2017, 103232N.  H. Durrant-Whyte, T.C. Henderson, Multisensor data fustion, in: B. Siciliano, O. Khatib (Eds.), Springer Handbook of Robotics, Springer, Berlin, 2016, pp. 867– 896.  W. Koch, On Bayesian tracking and data fusion: a tutorial introduction with examples, IEEE Aero. El. Sys. Mag. 25 (2010) 29–52.  K.O. Hill, Y. Tremblay, B.S. Kawasaki, Modal noise in multimode fiber links: theory and experiment, Appl. Opt. 5 (1980) 270–272.  L. Rodriguez-Cobo, M. Lomer, A. Cobo, J-M. Lopez-Higuera, Optical fiber strain sensor with extended dynamic range based on specklegrams, Sens. Actuat. APhys. 203 (2013) 341–345.  K. Peters, Polymer optical fiber sensors—a review, Smart Mater. Struct. 20 (2011) 013002.  E. Fujiwara, M.F.M. Santos, C.K. Suzuki, Optical fiber specklegram sensor analysis by speckle pattern division, Appl. Optics 56 (2017) 1585–1590.
 B. Redding, S.M. Popoff, H. Cao, All-fiber spectrometer based on speckle pattern reconstruction, Opt. Express 21 (2013) 6584–6600.  M. Ikeda, Y. Murakami, K. Kitayama, Mode scrambler for optical fibers, Appl. Opt. 16 (1977) 1045–1049.  T. Tsuji, T. Asakura, H. Fujii, Variation of the speckle contrast in a graded-index fibre by misalignment of the incident beam, Opt. Quant. Electron. 16 (1984), 9–18.
Eric Fujiwara was born in Sao Paulo, Brazil. He received the B.S. degree in control and automation engineering, and the M.S. and Ph.D. degrees in mechanical engineering from the University of Campinas (Unicamp), Brazil, in 2008, 2009, and 2012, respectively. Since 2014, he has been an Assistant Professor with the School of Mechanical Engineering, Unicamp. He is author of more than 40 journal and conference papers, and 7 inventions. His research interest includes optical fiber sensors, human-robot interaction, physical and chemical sensing, and fabrication of silica nanoparticles, silica glasses, and specialty optical fibers.
Yu Tzu Wu was born in Taipei, Taiwan. She received the B.S. degree in control and automation engineering from the University of Campinas (Unicamp), Brazil, in 2015. Since 2012, she has been with the Laboratory of Photonic Materials and Devices, Unicamp. Her research interest includes the development of mechatronics systems for applications in human-robot interaction and image processing.
Murilo Ferreira Marques dos Santos was born in Rio Claro, Brazil. He received the B.S. and M.S. degrees in mechanical engineering from the University of Campinas (Unicamp), Brazil, in 2012 and 2014, respectively. Since 2009, he has been with the
Laboratory of Photonic Materials and Devices, Unicamp. His research interests include the development and application of SiO2-based photonic materials.
Egont Alexandre Schenkel was born in Campinas, Brazil. He received the B.S. degree in physics and the M.S. degree in mechanical engineering from the University of Campinas (Unicamp), Brazil, in 2010 and 2015, respectively. Since 2013, he has been with the Laboratory of Photonic Materials and Devices, Unicamp. His research interests include the development and characterization of SiO2-based photonic materials.
Carlos Kenichi Suzuki was born in Sao Paulo, Brazil. He received the B.S. degree in physics from the University of Sao Paulo, Brazil, in 1969, the M.S. degree in physics from the University of Campinas (Unicamp), Brazil, in 1974, and the Ph.D. degree in applied physics engineering from the University of Tokyo, Japan, in 1981. Since 1994, he has been a Professor of Materials Engineering in the area of optical materials, optical fiber sensors, and nanotechnology with Unicamp. He has authored more than 160 journal and conference papers, and holds over 15 patents.